Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Checkpoint target lag metrics may leak after collection being released #39840

Closed
1 task done
congqixia opened this issue Feb 13, 2025 · 0 comments
Closed
1 task done

[Bug]: Checkpoint target lag metrics may leak after collection being released #39840

congqixia opened this issue Feb 13, 2025 · 0 comments
Assignees
@congqixia
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@congqixia
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: 2.5.4 & 2.4.21
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Checkpoint target lag metrics may leak after collection being released

Expected Behavior

this metrics shall be cleaned and not updated after collection released

Steps To Reproduce



Milvus Log


No response


Anything else?


No response

      		
    

      
  





        



            

              
            

        

      


      
    








      

    



      

  
            

    

      
    

    


      

          @congqixia
congqixia




            added
      

  kind/bug

  Issues or changes related a bug

      

  triage/accepted

  Indicates an issue or PR is ready to be actively worked on.

  labels


      Feb 13, 2025

    

  



    

    

      
    

    


      

          @congqixia
congqixia




              self-assigned this


      Feb 13, 2025

    

  



  


  
    
  
  

    
  congqixia 

      added a commit
        to congqixia/milvus
      that referenced
      this issue

    
      Feb 13, 2025
    
    

      


  

    

      


  

      
        @congqixia
  





      

        
          fix: Remove dispatcher tasks before cleaning up targets
        

          
            
          

      


      

          

    
    
    

  

    


      


      

          

    
    
    

  

  

    
  



      


      
      

        
          645a9b0
        
      

    

  

    

      
Related to milvus-io#39840

The target could be updated async in previous code. This PR make remove
collection from target observer block until all tasks related in
dispatchers are removed preventing the metrics being updated after
collection released.

Signed-off-by: Congqi Xia <[email protected]>

    




    

  




  

  

    
  


  

      

    @congqixia
congqixia



    mentioned this issue
    
      Feb 13, 2025
    





  


  

      
   Merged

  







  




  


  
    
  
  

    
  congqixia 

      added a commit
        to congqixia/milvus
      that referenced
      this issue

    
      Feb 13, 2025
    
    

      


  

    

      


  

      
        @congqixia
  





      

        
          fix: Remove dispatcher tasks before cleaning up targets
        

          
            
          

      


      

          

    
    
    

  

    


      


      

          

    
    
    

  

  

    
  



      


      
      

        
          6273d3b
        
      

    

  

    

      
Related to milvus-io#39840

The target could be updated async in previous code. This PR make remove
collection from target observer block until all tasks related in
dispatchers are removed preventing the metrics being updated after
collection released.

Signed-off-by: Congqi Xia <[email protected]>

    




    

  




  

  

    
  


  

      

    @congqixia
congqixia



    mentioned this issue
    
      Feb 13, 2025
    





  


  

      
   Merged

  







  




  


  
    
  
  

    
  congqixia 

      added a commit
        to congqixia/milvus
      that referenced
      this issue

    
      Feb 13, 2025
    
    

      


  

    

      


  

      
        @congqixia
  





      

        
          fix: Remove dispatcher tasks before cleaning up targets
        

          
            
          

      


      

          

    
    
    

  

    


      


      

          

    
    
    

  

  

    
  



      


      
      

        
          79f45dc
        
      

    

  

    

      
Related to milvus-io#39840

The target could be updated async in previous code. This PR make remove
collection from target observer block until all tasks related in
dispatchers are removed preventing the metrics being updated after
collection released.

Signed-off-by: Congqi Xia <[email protected]>

    




    

  




  

  

    
  


  

      

    @congqixia
congqixia



    mentioned this issue
    
      Feb 13, 2025
    





  


  

      
   Merged

  







  




  


  
    
  
  

    
  sre-ci-robot 

      pushed a commit
      that referenced
      this issue

    
      Feb 13, 2025
    
    

      


  

    

      


  

      
        @congqixia
  





      

        
          fix: [2.4] Check collection released before target checks (#39842)
        

          
            
          

      


      

          

    
    
    

  

    


      


      

          

    
    
    

  

  

    
  



      


      
      

        
          7a6ccd6
        
      

    

  

    

      
Cherry-pick from master
pr: #39841
Related to #39840

The target could be updated async in previous code. This PR make remove
collection from target observer block until all tasks related in
dispatchers are removed preventing the metrics being updated after
collection released.

---------

Signed-off-by: Congqi Xia <[email protected]>

    




    

  




  


  
    
  
  

    
  sre-ci-robot 

      pushed a commit
      that referenced
      this issue

    
      Feb 13, 2025
    
    

      


  

    

      


  

      
        @congqixia
  





      

        
          fix: [2.5] Check collection released before target checks (#39843)
        

          
            
          

      


      

          

    
    
    

  

    


      


      

          

    
    
    

  

  

    
  



      


      
      

        
          9407a3c
        
      

    

  

    

      
Cherry-pick from master
pr: #39841 
Related to #39840

The target could be updated async in previous code. This PR make remove
collection from target observer block until all tasks related in
dispatchers are removed preventing the metrics being updated after
collection released.

---------

Signed-off-by: Congqi Xia <[email protected]>

    




    

  




  


  
    
  
  

    
  sre-ci-robot 

      pushed a commit
      that referenced
      this issue

    
      Feb 14, 2025
    
    

      


  

    

      


  

      
        @congqixia
  





      

        
          fix: Check collection released before target checks (#39841)
        

          
            
          

      


      

          

    
    
    

  

    


      


      

          

    
    
    

  

  

    
  



      


      
      

        
          58045a3
        
      

    

  

    

      
Related to #39840

The target could be updated async in previous code. This PR make remove
collection from target observer block until all tasks related in
dispatchers are removed preventing the metrics being updated after
collection released.

---------

Signed-off-by: Congqi Xia <[email protected]>

    




    

  




  


  
    
  
  

    
  codingjaguar 

      pushed a commit
        to codingjaguar/milvus
      that referenced
      this issue

    
      Feb 14, 2025
    
    

      


  

    

      


  

      
        @congqixia
      
        @codingjaguar
  





      

        
          fix: Check collection released before target checks (milvus-io#39841)
        

          
            
          

      


      

          

    
    
    

  

    


      


      

      


      
      

        
          bd75cff
        
      

    

  

    

      
Related to milvus-io#39840

The target could be updated async in previous code. This PR make remove
collection from target observer block until all tasks related in
dispatchers are removed preventing the metrics being updated after
collection released.

---------

Signed-off-by: Congqi Xia <[email protected]>

    




    

  




    

    











  
  

    

      

  





  




  
  

              

  
    Sign up for free
    to join this conversation on GitHub.
    Already have an account?
    Sign in to comment


  



  






  
                    




    

  


      
  

    Assignees
  



        

      

        

  
      @congqixia
  
    congqixia
    

      








      


  


  

    Labels
  



    

      

    kind/bug

  Issues or changes related a bug
  

    triage/accepted

  Indicates an issue or PR is ready to be actively worked on.








      

  

    

        

    Projects
  


        




    None yet




  



      


  

    
  

    Milestone
  


      No milestone





    
      
          


  

    
      

        
    

    Development
  




          
    
No branches or pull requests







    
  




      

    

  
      

  

    

      1 participant
    

    

        
          @congqixia