Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(NSC): handle endpoint slice ready nil #1583

Merged
merged 1 commit into from
Dec 8, 2023

Conversation

aauren
Copy link
Collaborator

@aauren aauren commented Dec 8, 2023

@mrueg

In some cases it is possible for Endpoint.Conditions.Ready to be nil during the early stages of initialization. When this happens it causes kube-router to segfault. This fix tests for nil before testing for Ready.

Error looks like:

2023-12-08T19:26:16.577929290Z E1208 19:26:16.577532       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)                                                                    
2023-12-08T19:26:16.577944838Z goroutine 257 [running]:                                                                                                                                                                                                                                 
2023-12-08T19:26:16.577949087Z k8s.io/apimachinery/pkg/util/runtime.logPanic({0x2d4e640, 0x45f71d0})                                                                                                                                                                                    
2023-12-08T19:26:16.577957166Z  /home/shadowcat/go-workspace/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:75 +0x105                                                                                                                                                  
2023-12-08T19:26:16.577960142Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x0})                                                                                                                                                                                        
2023-12-08T19:26:16.577964279Z  /home/shadowcat/go-workspace/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:49 +0xbb                                                                                                                                                   
2023-12-08T19:26:16.577967687Z panic({0x2d4e640, 0x45f71d0})                                                                                                                                                                                                                            
2023-12-08T19:26:16.577971478Z  /home/shadowcat/sdk/go1.20.7/src/runtime/panic.go:890 +0x262                                                                                                                                                                                            
2023-12-08T19:26:16.577975270Z github.com/cloudnativelabs/kube-router/v2/pkg/controllers/proxy.(*NetworkServicesController).buildEndpointSliceInfo(0xc0004d43c0)                                                                                                                        
2023-12-08T19:26:16.577978699Z  /home/shadowcat/git-repos/kube-router/pkg/controllers/proxy/network_services_controller.go:1053 +0x4fb                                                                                                                                                  
2023-12-08T19:26:16.577982383Z github.com/cloudnativelabs/kube-router/v2/pkg/controllers/proxy.(*NetworkServicesController).OnEndpointsUpdate(0xc0004d43c0, 0xc001062000)                                                                                                               
2023-12-08T19:26:16.577985971Z  /home/shadowcat/git-repos/kube-router/pkg/controllers/proxy/network_services_controller.go:806 +0x688                                                                                                                                                   
2023-12-08T19:26:16.577989751Z github.com/cloudnativelabs/kube-router/v2/pkg/controllers/proxy.(*NetworkServicesController).handleEndpointSliceAdd(0xc0004d43c0, {0x3061b00, 0xc001062000})                                                                                             
2023-12-08T19:26:16.577992907Z  /home/shadowcat/git-repos/kube-router/pkg/controllers/proxy/network_services_controller.go:1780 +0xa6                                                                                                                                                   
2023-12-08T19:26:16.577996184Z github.com/cloudnativelabs/kube-router/v2/pkg/controllers/proxy.(*NetworkServicesController).newEndpointSliceEventHandler.func1({0x3061b00, 0xc001062000})                                                                                               
2023-12-08T19:26:16.577999770Z  /home/shadowcat/git-repos/kube-router/pkg/controllers/proxy/network_services_controller.go:1762 +0x36                                                                                                                                                   
2023-12-08T19:26:16.578003232Z k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd({0xc0008fcd50, 0xc0008fcd60, 0xc0008fcd70}, {0x3061b00, 0xc001062000}, 0x0)                                                                                                                 
2023-12-08T19:26:16.578006524Z  /home/shadowcat/go-workspace/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:239 +0x50                                                                                                                                                       
2023-12-08T19:26:16.578009873Z k8s.io/client-go/tools/cache.(*processorListener).run.func1()                                                                                                                                                                                            
2023-12-08T19:26:16.578013183Z  /home/shadowcat/go-workspace/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:974 +0x264                                                                                                                                                 
2023-12-08T19:26:16.578016541Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0009a4f08)                                                                                                                                                                                       
2023-12-08T19:26:16.578020098Z  /home/shadowcat/go-workspace/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x3f                                                                                                                                                     
2023-12-08T19:26:16.578023524Z k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00056ff08, {0x35f5940, 0xc00077d950}, 0x1, 0xc0006f6900)                                                                                                                                               
2023-12-08T19:26:16.578026871Z  /home/shadowcat/go-workspace/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xc7                                                                                                                                                     
2023-12-08T19:26:16.578030174Z k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0009a4f08, 0x3b9aca00, 0x0, 0x1, 0xc0006f6900)                                                                                                                                                          
2023-12-08T19:26:16.578033409Z  /home/shadowcat/go-workspace/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x85                                                                                                                                                     
2023-12-08T19:26:16.578036799Z k8s.io/apimachinery/pkg/util/wait.Until(0xc0009a4f08, 0x3b9aca00, 0xc0006f6900)                                                                                                                                                                          
2023-12-08T19:26:16.578040052Z  /home/shadowcat/go-workspace/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161 +0x33                                                                                                                                                     
2023-12-08T19:26:16.578043306Z k8s.io/client-go/tools/cache.(*processorListener).run(0xc000958750)                                                                                                                                                                                      
2023-12-08T19:26:16.578046528Z  /home/shadowcat/go-workspace/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:968 +0x7a                                                                                                                                                  
2023-12-08T19:26:16.578049920Z k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()                                                                                                                                                                                                 
2023-12-08T19:26:16.578053604Z  /home/shadowcat/go-workspace/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x90                                                                                                                                                         
2023-12-08T19:26:16.578057104Z created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start                                                                                                                                                                                              
2023-12-08T19:26:16.578064118Z  /home/shadowcat/go-workspace/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0xb1                                                                                                                                                         
2023-12-08T19:26:16.583308299Z panic: runtime error: invalid memory address or nil pointer dereference [recovered]                                                                                                                                                                      
2023-12-08T19:26:16.583322624Z  panic: runtime error: invalid memory address or nil pointer dereference                                                                                                                                                                                 
2023-12-08T19:26:16.583326297Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x23bfd5b]

In some cases it is possible for Endpoint.Conditions.Ready to be nil
during the early stages of initialization. When this happens it causes
kube-router to segfault. This fix tests for nil before testing for
Ready.
@aauren aauren merged commit aebaa48 into master Dec 8, 2023
7 checks passed
@aauren aauren deleted the fix_endpoint_slice_nil_pointer branch December 8, 2023 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant