add generate-package-code #167

chyroc · 2018-03-26T07:10:01Z

No description provided.

codecov-io · 2018-03-26T07:12:03Z

Codecov Report

Merging #167 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #167   +/-   ##
=======================================
  Coverage   81.72%   81.72%           
=======================================
  Files           8        8           
  Lines        1882     1882           
=======================================
  Hits         1538     1538           
  Misses        257      257           
  Partials       87       87

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9bd0532...efaebbe. Read the comment docs.

chyroc · 2018-03-26T07:14:26Z

At present, we only take bytes as an example of pr. If you think ok, I will add other packages.

chyroc · 2018-03-26T10:11:29Z

No, I mean maybe these standard packages can be done by generating code, because each package may have updates, and it is considered difficult to detect and update the code.
Then I just give the generated code and use bytes as the sample code. If I can, I will use the code to generate the code under the wrong package.

MichaelS11 · 2018-03-26T10:20:44Z

I now see what this is doing, very cool. The concept is great. :)

I am not sure if it will work in all cases. In particular, packages that have new methods adding to them, like sort as an example. There is a sort 1.8 build. Also note that some packages have appengine build constraints/tags.

It should at least be able to use as a good starting template for checking what we have or adding new packages.

mattn · 2018-03-26T10:28:48Z

It is difficult to create a portable package file using this tool. Because syscall has different methods depending on OS. So I think that it must have a dictionary or whitelist.

chyroc · 2018-03-26T11:18:27Z

can use makefile to make only specific packages use the generated code

MichaelS11 · 2018-03-26T14:42:58Z

Might be better for how it is now to remove the makefile and just run it manually as wanted. Great tool, can do lots with it, just not full automatic.

MichaelS11 · 2018-03-28T04:52:54Z

@chyroc
Could you please fix bytes?

chyroc · 2018-03-28T05:08:21Z

I have resolved the conflict, but there are still some problems.

Columns such as

	json.Number // type Number string
	time.Nanosecond // const Nanosecond  Duration = 1
	cookiejar.PublicSuffixList // type PublicSuffixList interface { }

are a type(but is not a struct) or a var or interface, so they cannot use AAA{}. At the moment I have added them to the blacklist, but I don't think this is a good way. Maybe we should support this type(but is not a struct) / var / interface, but what should we do?

MichaelS11 · 2018-03-28T06:43:58Z

time.Nanosecond is already added:
https://github.com/mattn/anko/blob/master/packages/time.go

MichaelS11 · 2018-03-28T06:58:36Z

I think something like this will work:
PackageTypes["encoding/json"] = map[string]interface{}{
"Number": json.Number(""),
}

Also can check out Signal:
https://github.com/mattn/anko/blob/master/packages/os.go

MichaelS11 · 2018-03-28T07:08:34Z

Note that the build tests are failing because json.Valid was not added until go 1.9

MichaelS11 · 2018-03-31T12:37:22Z

@chyroc
Just wanted to double check, you know this PR cannot be merged because the build is failing, right?

chyroc · 2018-04-03T16:06:43Z

I noticed, I will fix him, I'm sorry to be busy recently

chyroc · 2018-04-08T14:55:12Z

I have a problem: The type of import is now through:
1, bytes.Buffer{} imported in this way,
2, the type of non-structure is imported in this way

var signal os.Signal
reflect.TypeOf(&signal).Elem()

Whether to continue this way, or unified as the second way

mattn · 2018-04-08T15:00:12Z

interface can be nil. For example, This is typed nil.

v := io.Writer(nil)

MichaelS11 · 2018-04-08T17:14:12Z

A note, I think this code should try to handle all the cases. This code is great for generating a template that can be updated by hand.

chyroc · 2018-04-09T04:36:12Z

how to deal with math.MaxUint64, it return constant 18446744073709551615 overflows int in my machine

MichaelS11 · 2018-04-09T11:02:21Z

Should be able to just convert it to a uint64.

golang/go#19621
golang/go#23086

MichaelS11 · 2018-04-09T11:14:14Z

Have you read up on Build Constraints?

https://golang.org/pkg/go/build/

Looks like there are extra Build Constraints that are not wanted.

For example:

// +build go1.8,!go1.10

Which means build if (1.8 or greater) OR (not 1.10 or great), which probably not what we are going for.

MichaelS11 · 2018-04-09T11:16:25Z

Also there is the custom build tag appengine which I believe is used to create the engine for the web site. So // +build !appengine needs to be accounted for.

MichaelS11 · 2018-04-09T11:22:02Z

packages/errors.go

@@ -8,4 +8,6 @@ func init() {
 	Packages["errors"] = map[string]interface{}{
 		"New": errors.New,
 	}
+	PackageTypes["errors"] = map[string]interface{}{}


Empty PackageTypes can be removed.

MichaelS11 · 2018-04-09T11:23:16Z

packages/math.110.go

+func init_math() {
+
+	Packages["math"]["Erfcinv"] = math.Erfcinv
+


Not a big deal, but would be nice to remove the extra spaces.

MichaelS11 · 2018-04-09T11:28:58Z

packages/os.go

 		"ErrExist":          os.ErrExist,
 		"ErrInvalid":        os.ErrInvalid,
 		"ErrNotExist":       os.ErrNotExist,
 		"ErrPermission":     os.ErrPermission,
+		"Executable":        os.Executable,


Some of the added function should be added to // +build !appengine like Executable, Unsetenv, and etc.

Ok. Then just ask, how to know the list of // +build !appengine

MichaelS11 · 2018-04-09T11:31:38Z

packages/sort.go

 		"Search":            sort.Search,
 		"SearchFloat64s":    sort.SearchFloat64s,
 		"SearchInts":        sort.SearchInts,
 		"SearchStrings":     sort.SearchStrings,
+		"Slice":             sort.Slice,


Slice* should be added to // +build go1.8

chyroc · 2018-04-09T11:33:49Z

Yes, I read ~
The comma means and and the space means or, so the // +build go1.8,!go1.10 meaning here is 1.8 and above, but does not include 1.10

For example, the function init_math:
// +build go1.8,!go1.10 will build under 1.8/1.9 in file packages/math.18.go
// +build go1.10 will build under 1.10 in file packages/math.110.go
and they called by file packages/math.go

Take the function init_encoding_json for example:
// +build go1.8,!go1.9,!go1.10 will build under 1.8 in file packages/encoding.json.18.go
// +build go1.9 go1.10 will build under 1.9/1.10 in file packages/encoding.json.19.110.go
and they called by file packages/encoding.json.go

MichaelS11 · 2018-04-09T11:34:00Z

Also, how are you checking what version each of the newly adding functions and types are in?

MichaelS11 · 2018-04-09T11:37:31Z

Looking at json.Valid, that should be // +build go1.9 and the other file should be // +build !go1.9.

chyroc · 2018-04-09T11:37:33Z

Maybe we can run ./cmd/generate-std-packages/run.sh in the ci test. If there is a change in the code, there is a new file, and ci should fail.
Of course, it can also be performed manually on a regular basis (so as not to block other people's pr)

MichaelS11 · 2018-04-09T11:40:18Z

While we are only testing 1.8, 1.9, 1.10 in travis, still want to try to be compatible to lower versions when/if possible. That means trying to figure out which version functions and types were added, for all the newly added functions and types.

MichaelS11 · 2018-04-09T11:42:58Z

Note #49 for overflows int issue.

chyroc · 2018-04-09T15:03:43Z

Should we first create maps of all packages and then add functions or variables one by one.
then add

l.Lock()
if Packages["name"] = nil {
Packages["name"]  = make(map[string]interface{})
}
l.Unlock()

This avoids the need to create two file builds similar to the appengine environment

MichaelS11 · 2018-04-09T20:01:45Z

We are adding functions or variables one at a time, don't think a lock is needed. Not seeing how this avoids the need for build files.

If you want to have only have two appengine build files, can have those files sets a flag, then use that flag to set the data.

chyroc · 2018-06-15T07:56:25Z

run such code to generate code:

#!/bin/bash

mkdir go && cd go

for i in 1.6.4 1.7.6 1.8.7 1.9.6 1.10.3
do
   echo "download go $i ing."
   wget https://dl.google.com/go/go$i.linux-amd64.tar.gz && tar zxvf go$i.linux-amd64.tar.gz && mv go $i
   rm go$i.linux-amd64.tar.gz
   echo "done."
done

cd ..

find packages | grep -vE '_test|Engine' | grep '/' | xargs rm

go run cmd/generate-std-packages/*.go -go16 ./go/1.6.4/bin/go -go17 ./go/1.7.6/bin/go -go18 ./go/1.8.7/bin/go -go19 ./go/1.9.6/bin/go -go110 ./1.10.3/bin/go && gofmt -s -w ./packages/*

chyroc · 2018-06-15T08:00:00Z

@MichaelS11 @mattn please see again, thank you.

MichaelS11 · 2018-06-15T19:09:52Z

I leave this to @mattn

I was fine with the simple starting template and making changes by hand.

mattn · 2018-06-16T02:17:36Z

I don't have strong opnion, Is it possible to make composite literal like?

Packages["bytes"] = map[string]interface{}{
   ...
}

chyroc · 2018-06-16T13:59:21Z

should i add // Code generated by cmd/generate-std-packages. DO NOT EDIT. on the head of file

chyroc force-pushed the add/generate-package-code branch from 40b4b97 to 56a5ee5 Compare April 9, 2018 07:27

MichaelS11 suggested changes Apr 9, 2018

View reviewed changes

chyroc force-pushed the add/generate-package-code branch from 76a4572 to f993ed4 Compare April 9, 2018 14:48

chyroc force-pushed the add/generate-package-code branch from f993ed4 to 4f7d68f Compare April 9, 2018 15:18

chyroc force-pushed the add/generate-package-code branch 5 times, most recently from a87070e to 9bd7bd3 Compare June 15, 2018 07:39

add generate-package-code

efaebbe

chyroc force-pushed the add/generate-package-code branch from 9bd7bd3 to efaebbe Compare June 15, 2018 07:42

chyroc closed this Jul 27, 2018

chyroc deleted the add/generate-package-code branch July 27, 2018 12:12

		func init_math() {

		Packages["math"]["Erfcinv"] = math.Erfcinv

add generate-package-code #167

add generate-package-code #167

Conversation

chyroc commented Mar 26, 2018

codecov-io commented Mar 26, 2018 • edited Loading

Codecov Report

chyroc commented Mar 26, 2018

chyroc commented Mar 26, 2018

MichaelS11 commented Mar 26, 2018

mattn commented Mar 26, 2018

chyroc commented Mar 26, 2018

MichaelS11 commented Mar 26, 2018

MichaelS11 commented Mar 28, 2018

chyroc commented Mar 28, 2018 • edited Loading

MichaelS11 commented Mar 28, 2018

MichaelS11 commented Mar 28, 2018

MichaelS11 commented Mar 28, 2018

MichaelS11 commented Mar 31, 2018

chyroc commented Apr 3, 2018

chyroc commented Apr 8, 2018

mattn commented Apr 8, 2018

MichaelS11 commented Apr 8, 2018

chyroc commented Apr 9, 2018

MichaelS11 commented Apr 9, 2018

MichaelS11 commented Apr 9, 2018

MichaelS11 commented Apr 9, 2018

MichaelS11 Apr 9, 2018

Choose a reason for hiding this comment

chyroc Apr 9, 2018

Choose a reason for hiding this comment

MichaelS11 Apr 9, 2018

Choose a reason for hiding this comment

MichaelS11 Apr 9, 2018

Choose a reason for hiding this comment

chyroc Apr 9, 2018

Choose a reason for hiding this comment

MichaelS11 Apr 9, 2018

Choose a reason for hiding this comment

chyroc commented Apr 9, 2018

MichaelS11 commented Apr 9, 2018

MichaelS11 commented Apr 9, 2018

chyroc commented Apr 9, 2018

MichaelS11 commented Apr 9, 2018

MichaelS11 commented Apr 9, 2018

chyroc commented Apr 9, 2018

MichaelS11 commented Apr 9, 2018 • edited Loading

chyroc commented Jun 15, 2018

chyroc commented Jun 15, 2018

MichaelS11 commented Jun 15, 2018

mattn commented Jun 16, 2018

chyroc commented Jun 16, 2018

codecov-io commented Mar 26, 2018 •

edited

Loading

chyroc commented Mar 28, 2018 •

edited

Loading

MichaelS11 commented Apr 9, 2018 •

edited

Loading