io.Copy gives permission denied for get operation and io.ReadAll gives Bad Message for 190kb+ files

[aashakabra]

Hello,

I am doing a simple get operation on a file available on SFTP, but I am not writing the content in a file, I want to hold it in memory and want to pass it as output for futher transformation.

Below code works perfectly fine on our test SFTP server (openssh) but it fails with error on customer's SFTP server (which is very much older openssh and there are permission restrictions , I will list them out in detail).

OS details of sftp server  - Red Hat Enterprise Linux Server release 7.9 (Maipo)

Type of sftp server - MessageWay SFTP Interface Version 6.1, sftp server type : openSSH
OpenBSD: sshd_config,v 1.73
sftp> version : SFTP protocol version 3

Permissions on parent directory & file -  View message , upload message and download message (with other required permissions)
Permissions assigned to user - View message , upload message and download message (with other required permissions)
Restrictions applied on sftp server -  Whitelist public ip to establish connection towards access point.

The code implementation I have is-

// ProcessDataGetOperation downloads base64 binary encoded file content from SFTP server
func ProcessDataGetOperation(sc *sftp.Client, inputParams *InputParam, binary bool, activityName string) (*Output, error) {
	srcFile, err := sc.OpenFile(inputParams.RemoteFileName, (os.O_RDONLY))
	if err != nil {
		return nil, GetError(RemoteFileOpenError, activityName, err.Error())
	}
	defer srcFile.Close()

	// read the content of file hosted on sftp server
	dataBytes := new(bytes.Buffer)
	bytes, err := io.Copy(dataBytes, srcFile)

	//bytes, err := io.ReadAll(srcFile)
	if err != nil {
		return nil, GetError(GetOperationError, activityName, err.Error())
	}
	var fileInfo fileInfo
	fileInfo.FileName = inputParams.RemoteFileName
	fileInfo.NumberOfBytes = bytes

	var output Output
	if binary {
		output.BinaryData = base64.StdEncoding.EncodeToString(dataBytes.Bytes())
	} else {
		output.ASCIIData = dataBytes.String()
	}
	output.FileTransferred = append(output.FileTransferred, &fileInfo)

	return &output, nil
}

With this code we are getting below error on customer SFTP server- Permission denied
The permission detail that customer has highlighted to us is - The file on sftp server has read permission. The directory has read & execute permissions.
However Customer is able to drag and download the file using WinSCP.

I tried the same permission on my local openssh server but I did not face any issue.

Now , when I replace io.Copy with io.ReadAll - the code works fine but for file size less than 190kb. io.ReadAll fails for filesize greater than 190kb and error received is "Bad Message"(SSH_FX_BAD_MESSAGE)

What could be the possible reason to run into this issue? and can you suggest some solution alternate to io.Copy which could possibly fix the issue?

[aashakabra]

Hello @puellanivis , Can you please share some inputs around this?

Customer is able to get the content of a file on SFTP server for their another test instance. It is just above described SFTP server that they run into issue, which seems to be their main SFTP server.

Can you please suggest what could be the possible reason for this error and suggest alternate that I can try instead of io.Copy or io.ReadAll ?

[drakkan]

@aashakabra please avoid mentioning people directly. I understand the issue is urgent for you but you are using open source software for free, you can't expect help as if we were your colleagues working for the same company, we try to help when possible, but we also have our commitments, you opened this issue less than 12 hours ago. Thanks for understanding

[aashakabra]

I understand. Sorry for the inconvenience.

[puellanivis]

What options are you specifying in your sftp.NewClient call?

[aashakabra]

I am not specifying any options.

// Create an SFTP client on top of the SSH connection
logCache.Infof("Creating new SFTP client")
client, err := sftp.NewClient(conn)
if err != nil {
	return nil, fmt.Errorf("failed to create SFTP client: %s", err.Error())
}

I close the client connection in another call back method Stop which is called when engine is stopped.

[puellanivis]

Can you try using sftp.UseConcurrentReads(false) ? I suspect the permission denied is originating from trying to stat the file. That should make the io.Copy and io.ReadAll equivalent. (That is, eventually produce a SSH_FX_BAD_MESSAGE)

[aashakabra]

Thankyou for your reply.

Okay. Should I also add - sftp.UseFstat(true) along with sftp.UseConcurrentReads(false) ?

As per above ping if this will further lead to SSH_X_BAD_MESSAGE - what can be probable solution for this? as on this specific SFTP server they are facing this error for file above 190kb.

In past during development we faced size issue with io.ReadAll (no error but sftp get file content + sftp put file content -> the file size dint match. And then we decided to use io.Copy() instead which solved this issue for bigger file size. We never ran into error. It is this customer SFTP server where they are getting error further with 190kb+ file size - SSH_FX_BAD_MESSAGE

[puellanivis]

I don’t know if the UseFstat(true) will help, but it should be tried in alternative to the UseConcurrentReads(false) as they solve different issues. It could be that fstat works but stat doesn’t, but it’s also probable that neither might work. So UseConcurrentReads(false) will turn off even attempting either.

I’m not entirely sure why you’re getting SSH_FX_BAD_MESSAGE, and was just code diving to see if I could see anything. We definitely don’t generate it, so it’s coming from the service on the other side. I’m unsure what it could possibly be complaining about, but I’m thinking it might be from attempting a too large read maybe. But we’re defaulting to the smallest size that all servers are supposed to support.

It’s interesting that io.ReadAll was giving inconsistent file sizes. 🤔 That’s probably diagnostically significant, but I’m not sure I know what could be wrong.

PS: Which version of this library are you using?

[aashakabra]

I am using version github.com/pkg/sftp v1.13.6

Let me set the flag UseConcurrentReads(false) and give it a try on customer envt.
Can you confirm, if this will stop us from reading two different files concurrently?

[puellanivis]

This does not stop you from reading two different files concurrently. It stops the library from concurrently issuing read requests. It’s kind of a last ditch effort to provide the highest compatibility, but also the slowest speeds.

[aashakabra]

We tried below two things on customer's envt-

1. UseConcurrentReads(false)
   file < 190kb : pass
   file > 190kb : Usecase failed at sc.OpenFile itself giving error - file does not exist. Customer confirmed that the file is present on SFTP server.

2. UseFstat(true)
   file < 190kb : Usecase failed at io.Copy with error - sftp: "Operation unsupported" (SSH_FX_OP_UNSUPPORTED)
   file > 190kb : Usecase failed at sc.OpenFile itself giving error - file does not exist. Customer confirmed that the file is present on SFTP

I am using sc.OpenFile(inputParams.RemoteFileName, (os.O_RDONLY))

Can you suggest if there is any permission we can give at customer's end so that stat function do not give error?
or what can we try further?

[aashakabra]

Below are the SFTP server details that customer is connecting to --

OS details of sftp server  - Red Hat Enterprise Linux Server release 7.9 (Maipo)

Type of sftp server - MessageWay SFTP Interface Version 6.1, sftp server type : openSSH
OpenBSD: sshd_config,v 1.73
sftp> version : SFTP protocol version 3

Permissions on parent directory & file -  View message , upload message and download message (with other required permissions)
Permissions assigned to user - View message , upload message and download message (with other required permissions)
Restrictions applied on sftp server -  Whitelist public ip to establish connection towards access point.

[puellanivis]

The “file does not exist” errors are extremely weird, because there’s no zero reason why this should be the case. I would recommend that they double check, and triple check the filename. There’s pretty much no way that this library could work for files <190 kB, but “file does not exist” on the open…

The UseFstat(true) results are interesting… the SSH_FXP_FSTAT is defined in the spec, and so everyone should be supporting it… very curious.

As a wild shot, could you try the dev-v2 branch? It’s a pretty big rewrite, but the client API at least should be mostly the same. (OpenFile does change to be more similar to os.OpenFile though. Though if all you’re doing is os.O_RDONLY then that’s just unwrapping what Open does for you.)

[puellanivis]

I’ve released a new version v1.13.7 could you try that one?

[aashakabra]

Hello , Let me work on sharing this with customer and test. Meanwhile, can you suggest what exact change is done in this version which is expected to work for above usecase?

[puellanivis]

I don’t know of any specific change that could fix your problem… 🤷‍♀️ I don’t even know what is causing the problem in the first place. So, it’s a bit of darts in the dark.