Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

csharp/IceGrid/simple failure on Ubuntu 24.04 #2813

Closed
bernardnormier opened this issue Sep 27, 2024 · 6 comments
Closed

csharp/IceGrid/simple failure on Ubuntu 24.04 #2813

bernardnormier opened this issue Sep 27, 2024 · 6 comments
Assignees
Labels
Milestone

Comments

@bernardnormier
Copy link
Member

From https://github.com/zeroc-ice/ice/actions/runs/11075786900/job/30777485537 (on main):

*** [67/304] Running csharp/IceGrid/simple tests ***
[ running without deployment test - 09/27/24 18:49:33 ]
- Config: ssl,mx
starting IceGrid registry Master... (/home/runner/work/ice/ice/cpp/bin/icegridregistry --Ice.Default.Host=127.0.0.1 --Ice.Warn.Connections=0 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Admin.Endpoints="tcp -h 127.0.0.1" --Ice.Admin.InstanceName=Server --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --IceSSL.*** --IceSSL.DefaultDir=/home/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.CertFile=server.p12 --Ice.PrintStackTraces=1 --IceGrid.InstanceName=TestIceGrid --IceGrid.Registry.PermissionsVerifier=TestIceGrid/NullPermissionsVerifier --IceGrid.Registry.AdminPermissionsVerifier=TestIceGrid/NullPermissionsVerifier --IceGrid.Registry.SSLPermissionsVerifier=TestIceGrid/NullSSLPermissionsVerifier --IceGrid.Registry.AdminSSLPermissionsVerifier=TestIceGrid/NullSSLPermissionsVerifier --IceGrid.Registry.Server.Endpoints=default --IceGrid.Registry.Internal.Endpoints=default --IceGrid.Registry.Client.Endpoints="default -p 14020" --IceGrid.Registry.Discovery.Port=14099 --IceGrid.Registry.SessionManager.Endpoints=default --IceGrid.Registry.AdminSessionManager.Endpoints=default --IceGrid.Registry.ReplicaName=Master --Ice.ProgramName=Master --Ice.PrintAdapterReady=1 --Ice.ThreadPool.Client.SizeWarn=0 --IceGrid.Registry.LMDB.MapSize=1 --IceGrid.Registry.LMDB.Path=/home/runner/work/ice/ice/csharp/test/IceGrid/simple/registry-Master --IceGrid.Registry.Client.ThreadPool.SizeWarn=0 --IceGrid.Registry.DefaultTemplates="/home/runner/work/ice/ice/cpp/config/templates.xml" --IceGrid.Registry.DynamicRegistration=1 --IceGrid.Registry.Trace.Discovery=2 --Ice.LogFile=/home/runner/work/ice/ice/csharp/test/IceGrid/simple/Master-092724-1849.log env={'LD_LIBRARY_PATH': '/home/runner/work/ice/ice/cpp/lib/x86_64-linux-gnu'})
ok
starting IceGrid registry Slave1... (/home/runner/work/ice/ice/cpp/bin/icegridregistry --Ice.Default.Host=127.0.0.1 --Ice.Warn.Connections=0 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Admin.Endpoints="tcp -h 127.0.0.1" --Ice.Admin.InstanceName=Server --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --IceSSL.*** --IceSSL.DefaultDir=/home/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.CertFile=server.p12 --Ice.PrintStackTraces=1 --IceGrid.InstanceName=TestIceGrid --IceGrid.Registry.PermissionsVerifier=TestIceGrid/NullPermissionsVerifier --IceGrid.Registry.AdminPermissionsVerifier=TestIceGrid/NullPermissionsVerifier --IceGrid.Registry.SSLPermissionsVerifier=TestIceGrid/NullSSLPermissionsVerifier --IceGrid.Registry.AdminSSLPermissionsVerifier=TestIceGrid/NullSSLPermissionsVerifier --IceGrid.Registry.Server.Endpoints=default --IceGrid.Registry.Internal.Endpoints=default --IceGrid.Registry.Client.Endpoints="default -p 14021" --IceGrid.Registry.Discovery.Port=14099 --IceGrid.Registry.SessionManager.Endpoints=default --IceGrid.Registry.AdminSessionManager.Endpoints=default --IceGrid.Registry.ReplicaName=Slave1 --Ice.ProgramName=Slave1 --Ice.PrintAdapterReady=1 --Ice.ThreadPool.Client.SizeWarn=0 --IceGrid.Registry.LMDB.MapSize=1 --IceGrid.Registry.LMDB.Path=/home/runner/work/ice/ice/csharp/test/IceGrid/simple/registry-Slave1 --IceGrid.Registry.Client.ThreadPool.SizeWarn=0 --IceGrid.Registry.DefaultTemplates="/home/runner/work/ice/ice/cpp/config/templates.xml" --Ice.Default.Locator="TestIceGrid/Locator:default -p 14020" --IceGrid.Registry.DynamicRegistration=1 --IceGrid.Registry.Trace.Discovery=2 --Ice.LogFile=/home/runner/work/ice/ice/csharp/test/IceGrid/simple/Slave1-092724-1849.log env={'LD_LIBRARY_PATH': '/home/runner/work/ice/ice/cpp/lib/x86_64-linux-gnu'})
ok
starting IceGrid registry Slave2... (/home/runner/work/ice/ice/cpp/bin/icegridregistry --Ice.Default.Host=127.0.0.1 --Ice.Warn.Connections=0 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Admin.Endpoints="tcp -h 127.0.0.1" --Ice.Admin.InstanceName=Server --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --IceSSL.*** --IceSSL.DefaultDir=/home/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.CertFile=server.p12 --Ice.PrintStackTraces=1 --IceGrid.InstanceName=TestIceGrid --IceGrid.Registry.PermissionsVerifier=TestIceGrid/NullPermissionsVerifier --IceGrid.Registry.AdminPermissionsVerifier=TestIceGrid/NullPermissionsVerifier --IceGrid.Registry.SSLPermissionsVerifier=TestIceGrid/NullSSLPermissionsVerifier --IceGrid.Registry.AdminSSLPermissionsVerifier=TestIceGrid/NullSSLPermissionsVerifier --IceGrid.Registry.Server.Endpoints=default --IceGrid.Registry.Internal.Endpoints=default --IceGrid.Registry.Client.Endpoints="default -p 14022" --IceGrid.Registry.Discovery.Port=14099 --IceGrid.Registry.SessionManager.Endpoints=default --IceGrid.Registry.AdminSessionManager.Endpoints=default --IceGrid.Registry.ReplicaName=Slave2 --Ice.ProgramName=Slave2 --Ice.PrintAdapterReady=1 --Ice.ThreadPool.Client.SizeWarn=0 --IceGrid.Registry.LMDB.MapSize=1 --IceGrid.Registry.LMDB.Path=/home/runner/work/ice/ice/csharp/test/IceGrid/simple/registry-Slave2 --IceGrid.Registry.Client.ThreadPool.SizeWarn=0 --IceGrid.Registry.DefaultTemplates="/home/runner/work/ice/ice/cpp/config/templates.xml" --Ice.Default.Locator="TestIceGrid/Locator:default -p 14020" --IceGrid.Registry.DynamicRegistration=1 --IceGrid.Registry.Trace.Discovery=2 --Ice.LogFile=/home/runner/work/ice/ice/csharp/test/IceGrid/simple/Slave2-092724-1849.log env={'LD_LIBRARY_PATH': '/home/runner/work/ice/ice/cpp/lib/x86_64-linux-gnu'})
ok
starting IceGrid node localnode... (/home/runner/work/ice/ice/cpp/bin/icegridnode --Ice.Default.Host=127.0.0.1 --Ice.Warn.Connections=1 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Admin.Endpoints="tcp -h 127.0.0.1" --Ice.Admin.InstanceName=Server --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --IceSSL.*** --IceSSL.DefaultDir=/home/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.CertFile=server.p12 --Ice.PrintStackTraces=1 --IceGrid.InstanceName=TestIceGrid --IceGrid.Node.Endpoints=default --IceGrid.Node.WaitTime=240 --Ice.ProgramName=icegridnode --IceGrid.Node.Trace.Replica=0 --IceGrid.Node.Trace.Activator=0 --IceGrid.Node.Trace.Adapter=0 --IceGrid.Node.Trace.Server=0 --IceGrid.Node.ThreadPool.SizeWarn=0 --IceGrid.Node.PrintServersReady=node --IceGrid.Node.Name=localnode --IceGrid.Node.Data=/home/runner/work/ice/ice/csharp/test/IceGrid/simple/node-localnode --IceGrid.Node.PropertiesOverride="Ice.Default.Host=127.0.0.1 Test.BasePort=14000 Ice.Warn.Connections=1 Ice.Default.Protocol=ssl Ice.IPv6=0 Ice.Admin.Endpoints=\"tcp -h 127.0.0.1\" Ice.Admin.InstanceName=Server IceMX.Metrics.Debug.GroupBy=id IceMX.Metrics.Parent.GroupBy=parent IceMX.Metrics.All.GroupBy=none IceSSL.*** IceSSL.DefaultDir=/home/runner/work/ice/ice/certs IceSSL.CAs=cacert.pem IceSSL.VerifyPeer=2 IceSSL.CertFile=server.p12 Ice.ThreadPool.Server.Size=1 Ice.ThreadPool.Server.SizeMax=3 Ice.ThreadPool.Server.SizeWarn=0 Ice.PrintAdapterReady=1" --Ice.Default.Locator="TestIceGrid/Locator:default -p 14020:default -p 14021:default -p 14022" env={'LD_LIBRARY_PATH': '/home/runner/work/ice/ice/cpp/lib/x86_64-linux-gnu'})
ok
(dotnet /home/runner/work/ice/ice/csharp/test/IceGrid/simple/msbuild/server/net8.0/server.dll --Ice.Default.Host=127.0.0.1 --Test.BasePort=14000 --Ice.Warn.Connections=1 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Admin.Endpoints="tcp -h 127.0.0.1" --Ice.Admin.InstanceName=Server --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --IceSSL.*** --IceSSL.DefaultDir=/home/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.VerifyPeer=2 --IceSSL.CertFile=server.p12 --Ice.ThreadPool.Server.Size=1 --Ice.ThreadPool.Server.SizeMax=3 --Ice.ThreadPool.Server.SizeWarn=0 --Ice.PrintAdapterReady=1 --Ice.Default.Locator="TestIceGrid/Locator:default -p 14020" --TestAdapter.Endpoints=default --TestAdapter.AdapterId=TestAdapter)
(dotnet /home/runner/work/ice/ice/csharp/test/IceGrid/simple/msbuild/client/net8.0/client.dll --Ice.Default.Host=127.0.0.1 --Test.BasePort=14000 --Ice.Warn.Connections=1 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Admin.Endpoints="tcp -h 127.0.0.1" --Ice.Admin.InstanceName=Client --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --IceSSL.*** --IceSSL.DefaultDir=/home/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.VerifyPeer=2 --IceSSL.CertFile=client.p12 --Ice.Default.Locator="TestIceGrid/Locator:default -p 14020" --IceLocatorDiscovery.Timeout=50 --IceLocatorDiscovery.RetryCount=5 --IceLocatorDiscovery.Interface= --IceLocatorDiscovery.Port=14099 --IceLocatorDiscovery.Trace.Lookup=3 --Ice.LogFile=/home/runner/work/ice/ice/csharp/test/IceGrid/simple/client-092724-1849.log)
testing stringToProxy... ok
testing IceGrid.Locator is present... ok
testing checked cast... ok
pinging server... ok
testing locator finder... ok
testing discovery... failed:
saved /home/runner/work/ice/ice/csharp/test/IceGrid/simple/client-092724-1849.log
-! 09/27/2024 18:49:36:754 server: warning: connection exception:
   Ice.ConnectionLostException: Exception of type 'Ice.ConnectionLostException' was thrown.
      at Ice.SSL.TransceiverI.finishRead(Buffer buf) in /home/runner/work/ice/ice/csharp/src/Ice/SSL/TransceiverI.cs:line 176
      at Ice.Internal.IdleTimeoutTransceiverDecorator.finishRead(Buffer buf) in /home/runner/work/ice/ice/csharp/src/Ice/Internal/IdleTimeoutTransceiverDecorator.cs:line 56
      at Ice.ConnectionI.finishAsync(Int32 operation) in /home/runner/work/ice/ice/csharp/src/Ice/ConnectionI.cs:line 716
   local address = 127.0.0.1:33053
   remote address = 127.0.0.1:58056
(/home/runner/work/ice/ice/cpp/bin/icegridadmin --Ice.Default.Host=127.0.0.1 --Ice.Warn.Connections=1 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Admin.Endpoints="tcp -h 127.0.0.1" --Ice.Admin.InstanceName=Client --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --IceSSL.*** --IceSSL.DefaultDir=/home/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.CertFile=client.p12 --Ice.PrintStackTraces=1 --Ice.Default.Locator="TestIceGrid/Locator:default -p 14020" --IceGridAdmin.Username=admin1 --IceGridAdmin.*** -r Master -e "node shutdown localnode" env={'LD_LIBRARY_PATH': '/home/runner/work/ice/ice/cpp/lib/x86_64-linux-gnu'})
(/home/runner/work/ice/ice/cpp/bin/icegridadmin --Ice.Default.Host=127.0.0.1 --Ice.Warn.Connections=1 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Admin.Endpoints="tcp -h 127.0.0.1" --Ice.Admin.InstanceName=Client --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --IceSSL.*** --IceSSL.DefaultDir=/home/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.CertFile=client.p12 --Ice.PrintStackTraces=1 --Ice.Default.Locator="TestIceGrid/Locator:default -p 14020" --IceGridAdmin.Username=admin1 --IceGridAdmin.*** -r Master -e "registry shutdown Master" env={'LD_LIBRARY_PATH': '/home/runner/work/ice/ice/cpp/lib/x86_64-linux-gnu'})
(/home/runner/work/ice/ice/cpp/bin/icegridadmin --Ice.Default.Host=127.0.0.1 --Ice.Warn.Connections=1 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Admin.Endpoints="tcp -h 127.0.0.1" --Ice.Admin.InstanceName=Client --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --IceSSL.*** --IceSSL.DefaultDir=/home/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.CertFile=client.p12 --Ice.PrintStackTraces=1 --Ice.Default.Locator="TestIceGrid/Locator:default -p 14021" --IceGridAdmin.Username=admin1 --IceGridAdmin.*** -r Slave1 -e "registry shutdown Slave1" env={'LD_LIBRARY_PATH': '/home/runner/work/ice/ice/cpp/lib/x86_64-linux-gnu'})
(/home/runner/work/ice/ice/cpp/bin/icegridadmin --Ice.Default.Host=127.0.0.1 --Ice.Warn.Connections=1 --Ice.Default.Protocol=ssl --Ice.IPv6=0 --Ice.Admin.Endpoints="tcp -h 127.0.0.1" --Ice.Admin.InstanceName=Client --IceMX.Metrics.Debug.GroupBy=id --IceMX.Metrics.Parent.GroupBy=parent --IceMX.Metrics.All.GroupBy=none --IceSSL.*** --IceSSL.DefaultDir=/home/runner/work/ice/ice/certs --IceSSL.CAs=cacert.pem --IceSSL.CertFile=client.p12 --Ice.PrintStackTraces=1 --Ice.Default.Locator="TestIceGrid/Locator:default -p 14022" --IceGridAdmin.Username=admin1 --IceGridAdmin.*** -r Slave2 -e "registry shutdown Slave2" env={'LD_LIBRARY_PATH': '/home/runner/work/ice/ice/cpp/lib/x86_64-linux-gnu'})
saved /home/runner/work/ice/ice/csharp/test/IceGrid/simple/Slave2-092724-1849.log
saved /home/runner/work/ice/ice/csharp/test/IceGrid/simple/Slave1-092724-1849.log
saved /home/runner/work/ice/ice/csharp/test/IceGrid/simple/Master-092724-1849.log

test in IceGrid/simple failed:
@bernardnormier bernardnormier added this to the 3.8.0 milestone Sep 27, 2024
@bernardnormier
Copy link
Member Author

@bernardnormier bernardnormier self-assigned this Oct 18, 2024
@pepone
Copy link
Member

pepone commented Nov 1, 2024

The client log shows and exception sending the datagram lookup requets

-- 10/17/2024 15:04:03:894 client: Lookup: retrying locator lookup:
   lookup = IceLocatorDiscovery/Lookup -d -e 1.1:udp -h 239.255.0.1 -p 14299 --interface 127.0.0.1:udp -h 239.255.0.1 -p 14299 --interface 10.1.0.17:udp -h 239.255.0.1 -p 14299 --interface 172.17.0.1
   retry count = 1
   instance name = unknown
-- 10/17/2024 15:04:03:997 client: Lookup: locator lookup timed out:
   lookup = IceLocatorDiscovery/Lookup -d -e 1.1:udp -h 239.255.0.1 -p 14299 --interface 127.0.0.1:udp -h 239.255.0.1 -p 14299 --interface 10.1.0.17:udp -h 239.255.0.1 -p 14299 --interface 172.17.0.1
   instance name = unknown
-- 10/17/2024 15:04:04:040 client: Lookup: looking up locator:
   lookup = IceLocatorDiscovery/Lookup -d -e 1.1:udp -h 239.255.0.1 -p 0 --interface unknown
-! 10/17/2024 15:04:04:058 client: warning: failed to lookup locator with lookup proxy `IceLocatorDiscovery/Lookup -d -e 1.1:udp -h 239.255.0.1 -p 0 --interface unknown':
   Ice.SocketException: Exception of type 'Ice.SocketException' was thrown.
    ---> System.ArgumentException: couldn't find interface `unknown'
      at Ice.Internal.Network.getInterfaceAddress(String iface, AddressFamily family) in /home/runner/work/ice/ice/csharp/src/Ice/Internal/Network.cs:line 1170
      at Ice.Internal.Network.setMcastInterface(Socket socket, String iface, AddressFamily family) in /home/runner/work/ice/ice/csharp/src/Ice/Internal/Network.cs:line 392
      --- End of inner exception stack trace ---
      at Ice.Internal.OutgoingAsyncBase.invokeException() in /home/runner/work/ice/ice/csharp/src/Ice/Internal/OutgoingAsync.cs:line 100
   --- End of stack trace from previous location ---
      at IceLocatorDiscovery.LocatorI.<invoke>g__preformFindLocatorAsync|5_0(LookupPrx lookupPrx, LookupReplyPrx lookupReplyPrx) in /home/runner/work/ice/ice/csharp/src/IceLocatorDiscovery/PluginI.cs:line 465
-- 10/17/2024 15:04:04:059 client: Lookup: locator lookup failed:
   lookup = IceLocatorDiscovery/Lookup -d -e 1.1:udp -h 239.255.0.1 -p 0 --interface unknown
   Ice.SocketException: Exception of type 'Ice.SocketException' was thrown.
    ---> System.ArgumentException: couldn't find interface `unknown'
      at Ice.Internal.Network.getInterfaceAddress(String iface, AddressFamily family) in /home/runner/work/ice/ice/csharp/src/Ice/Internal/Network.cs:line 1170
      at Ice.Internal.Network.setMcastInterface(Socket socket, String iface, AddressFamily family) in /home/runner/work/ice/ice/csharp/src/Ice/Internal/Network.cs:line 392
      --- End of inner exception stack trace ---
      at Ice.Internal.OutgoingAsyncBase.invokeException() in /home/runner/work/ice/ice/csharp/src/Ice/Internal/OutgoingAsync.cs:line 100
   --- End of stack trace from previous location ---
      at IceLocatorDiscovery.LocatorI.<invoke>g__preformFindLocatorAsync|5_0(LookupPrx lookupPrx, Looku

We actually set the unknown interface in the test.

@bernardnormier
Copy link
Member Author

Currently waiting for a new failure with more tracing. I can't reproduce it in a devcontainer.

@externl
Copy link
Member

externl commented Nov 11, 2024

@InsertCreativityHere
Copy link
Member

This isn't specific to Ubuntu apparently, and doesn't require ssl, mx.
I got this when testing locally on Windows 11, with normal tcp:

*** [137/256] Running csharp/IceGrid/simple tests ***
[ running without deployment test - 11/18/24 16:21:34 ]
- Config: Debug,x64
starting IceGrid registry Master... ok
starting IceGrid registry Slave1... ok
starting IceGrid registry Slave2... ok
starting IceGrid node localnode... ok
testing stringToProxy... ok
testing IceGrid.Locator is present... ok
testing checked cast... ok
pinging server... ok
testing locator finder... ok
testing discovery... failed:


saved D:\Code\Workspace\ice\csharp\test\IceGrid\simple\client-111824-1621.log
saved D:\Code\Workspace\ice\csharp\test\IceGrid\simple\server-111824-1621.log
saved D:\Code\Workspace\ice\csharp\test\IceGrid\simple\Slave2-111824-1621.log
saved D:\Code\Workspace\ice\csharp\test\IceGrid\simple\Slave1-111824-1621.log
saved D:\Code\Workspace\ice\csharp\test\IceGrid\simple\Master-111824-1621.log
unexpected exit status: expected: 0, got 1


test in IceGrid/simple failed:
unexpected exit status: expected: 0, got 1

Here are my logs:
IceGrid-Simple-Failure-Logs.zip

@bernardnormier
Copy link
Member Author

This failure is very difficult to reproduce and was probably fixed by #3177.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants