regen_perly.pl: remove mostly dead gather_tokens code

The idea behind this code was to find the big enum block in perlytmp.h
that looks like:

    enum yytokentype {
      YYEMPTY = -2,
      YYEOF = 0,                     /* "end of file"  */
      YYerror = 256,                 /* error  */
      YYUNDEF = 257,                 /* "invalid token"  */
      GRAMPROG = 258,                /* GRAMPROG  */
      GRAMEXPR = 259,                /* GRAMEXPR  */
      GRAMBLOCK = 260,               /* GRAMBLOCK  */
      ...
    };

... and append to it (in perly.h) a series of equivalent macro
definitions, one for each (non-negative) enum symbol:

    #define YYEOF 0
    #define YYerror 256
    #define YYUNDEF 257
    #define GRAMPROG 258
    #define GRAMEXPR 259
    #define GRAMBLOCK 260
    ...

However, due to slight formatting changes in the code generated by bison
3+, this code has been essentially dead because the starting regex
didn't match anymore, so $gather_tokens was never set to a true value.

The last time we had token macros in perly.h was before commit
a9f5ab8 (in 2016), in which I generated perly.h using bison 3 for
the first time (without noticing the disappearance of the token macros).

We could try to fix the regex logic in regen_perly.pl, but given that no
one has complained about the missing token macros in 8 years (as far as
I know), let's just remove the code. (Yay, accidental scream tests?)

regen_perly.pl

@@ -133,8 +133,6 @@
 }
 
 my $endcore_done = 0;
-my $gather_tokens = 0;
-my $tokens;
 while (<$tmph_fh>) {
     # bison 2.6 adds header guards, which break things because of where we
     # insert #ifdef PERL_CORE, so strip them because they aren't important
@@ -162,19 +160,6 @@
 	$endcore_done = 1;
     }
     next if /^#line \d+ ".*"/;
-    if (!$gather_tokens) {
-	$gather_tokens = 1 if /^\s* enum \s* yytokentype \s* \{/x;
-    }
-    else {
-	if (/^\# \s* endif/x) { # The #endif just after the end of the token enum
-	    $gather_tokens = 0;
-	    $_ .= "\n/* Tokens.  */\n$tokens";
-	}
-	else {
-	    my ($tok, $val) = /(\w+) \s* = \s* (\d+)/x;
-	    $tokens .= "#define $tok $val\n" if $tok;
-	}
-    }
     print $h_fh $_;
 }
 close $tmph_fh;