Spliting file and save it with date...

[Michelle Konzack [e-mail] [sed-users]]

Good day,

I have seleral 1000 records of wind/solar data which where inserted in a
HTML page manually and it looks like:

<TR class="table_energy_row">
<TD class="energytable_num" >1</TD>
<TD class="energytable_date" >2017-07-23 Sat</TD>
<TD class="energytable_time" >23:59</TD>
<TD class="energytable_wh_solar">1387</TD>
<TD class="energytable_wh_wind" >0</TD>
<TD class="energytable_comment" >Absolutely no wind the last 18 hours.<BR>Solarcharger went into FLOAT, because the battery was full.</TD>
</TR>

So, I made a script and now my records look like

<?php
$date = "2017-07-23";
$time = "23:59";
$wh_solar = "1387";
$wh_wind = "0";
$comment = "Absolutely no wind the last 18 hours.<BR>Solarcharger went into FLOAT, because the battery was full.";
<?php
$date = "2017-07-24";
$time = "23:59";
$wh_solar = "1125";
$wh_wind = "2";
$comment = "Only very little wind.";

what I need is, to split the file from <?php to the line before the next
one and save it with the date as e.g. 2017-07-23.data

How can I do this mos easyly?

Thanks in avance
Michelle

[ionut [e-mail] [sed-users]]

Hello, Should the file "2017-07-23.data" look like below, or do you want other cleanup as well? <?php$date = "2017-07-23";$time = "23:59";$wh_solar = "1387";$wh_wind = "0"; [Non-text portions of this message have been removed]

[Jim Hill [e-mail] [sed-users]]

This is one of those tasks sed turns into an easy oneliner.

The idiom I use for multiline gaggles with identifiable start lines is
`/start/!{H;$!d};x;1d`. sed scripts only continue past that with a full
gaggle in the pattern buffer. H is "append this line to the hold buffer
with a leading newline", x is "exchange this line with the hold buffer", d
is best pronounced here "drop this line for now". So on the first line,
you'll have the start-line pattern, it will be exchanged with the hold
buffer and then `1d` since it's the first line dropped. Subsequent lines
that don't match the start pattern will be `H` appended to the hold buffer
with a leading newline and `$!d` dropped if not eof. Subsequent start lines
(and the eof line) will be exchanged with the accumulated gaggle in the
hold buffer, leaving the line buffer with the accumulated gaggle where you
can have your way with it.

So for this,

sed '/^</!{H;$!d};x;1d;s/.*\n$date = "\([^"]*\).*/cat >\1.data
<<EOD\n&\nEOD/' thephpfile #|sh

will transform each gaggle into a command to dump its contents into a file
named for its date field, and if you remove the `#` will pipe the results
through the shell to actually do it.


[Non-text portions of this message have been removed]

[[e-mail] [sed-users]]

That looks really great, very useful and creative. But it does not seem to work for some reason. I can see what you are doing and it should work.

$ cat thephpfile # I simplified a little by removing wind and comment
<?php
$date = "2017-07-23";
$time = "23:59";
$wh_solar = "1387";
<?php
$date = "2017-07-24";
$time = "23:59";
$wh_solar = "1125";

$ sed '/^</!{H;$!d};x;1d;s/.*\n$date = "\([^"]*\).*/cat >\1.data <<EOD\n&\nEOD/' thephpfile > cmd2.sh

$ diff -s thephpfile cmd2.sh
Files thephpfile and cmd2.sh are identical

////////////////////////

So nothing happened, there was no transformation. :(

Looking a little more, the s command is not running, because the pattern does not quite match. The multiple spaces before the equals sign do not match. I think it needs to be (add * before =):

s/.*\n$date *= "\([^"]*\).*/cat >\1.data <<EOD\n&\nEOD/

$ sed '/^</!{H;$!d};x;1d;s/.*\n$date *= "\([^"]*\).*/cat >\1.data <<EOD\n&\nEOD/' thephpfile
cat >2017-07-23.data <<EOD
<?php
$date = "2017-07-23";
$time = "23:59";
$wh_solar = "1387";
EOD
cat >2017-07-24.data <<EOD
<?php
$date = "2017-07-24";
$time = "23:59";
$wh_solar = "1125";
EOD

/////////////////////

The other little problem is that "$date" will be treated as a variable, so the script will not work as expected, "$date" will end up blank.

$ cat 2017-07-23.data
<?php
= "2017-07-23";
= "23:59";
= "1387";

I know there is some way to make the "here document" ignore the substitution, by quoting EOD, or the dollar sign could be escaped in a separate step. Again, it is an incredibly useful method you explained.

Daniel




[Non-text portions of this message have been removed]

[Jim Hill [e-mail] [sed-users]]

Yes, that's correct, I see I didn't cc: the list when I sent the
EOD-quoting correction to the OP, oops^2.

The ` *` change is a definite improvement, thanks.

sed '/^</!{H;$!d};x;1d;s/.*\n$date *= *"\([^"]*\).*/cat >\1.data
<<\\EOD\n&\nEOD/' t.data

Is something like this not in the sed oneliners?


[Non-text portions of this message have been removed]

[[e-mail] [sed-users]]

Based on Jim's code, I shortened it a bit and, in case of GNU sed, removed the necessity to pipe the output through the shell, by using the 'e' command:

sed '/^</!{H;$!d};x;s/.*$date *= *"\([^"]*\).*/cat>\1.data<<\\@\n&\n@/e;d' file



The ';d' at the end is there only for aesthetic reasons, to avoid printing empty lines.






[Non-text portions of this message have been removed]

[Cameron Simpson [e-mail] [sed-users]]

Post by Jim Hill ***@gmail.com [sed-users]
Yes, that's correct, I see I didn't cc: the list when I sent the
EOD-quoting correction to the OP, oops^2.
The ` *` change is a definite improvement, thanks.
sed '/^</!{H;$!d};x;1d;s/.*\n$date *= *"\([^"]*\).*/cat >\1.data
<<\\EOD\n&\nEOD/' t.data
Is something like this not in the sed oneliners?

Because it is so dangerous. Doing it with the shell is just what I might
consider doing, with some big glaring cautions, sufficent to change approaches.

First, make it clear to the people trying to use this script that you _do_ need
to quote the data markers (the "EOD"). I see you'd done that, avoiding $subst
in the data. Good.

But second, and this is critical, you need to be absolutely totally sure that
the marker "EOD" does not appear in the source data.

If it does appear then (a) the data will terminate earlier (just annoying) but
(b) critically, the following data will be fed to your shell as commands.

So recipes like this are VERY VERY prone to injection attacks.

Now, you _could_ do some kind of transform on the data to escape in some sense
the "EOD" marker, and undo it in the shell script. But that is hazard prone in
that it is easy to get wrong.

Because of this I would probably sidestep sed and go straight to Python or the
like where I don't need to risk interpreting data as commands.

Cheers,
Cameron Simpson <***@cskk.id.au> (formerly ***@zip.com.au)

[[e-mail] [sed-users]]

"But second, and this is critical, you need to be absolutely totally sure that the marker "EOD" does not appear in the source data. If it does appear then (a) the data will terminate earlier (just annoying) but (b) critically, the following data will be fed to your shell as commands."


Respectfully, I do not think that is the case. I would like to see that (data terminate early, following data fed to shell as commands) in an example. To my understanding, there is no risk, because the EOD marker has to appear on a line by itself. The bash reference manual says "This type of redirection instructs the shell to read input from the current source until a line containing only word (with no trailing blanks) is seen." We know the format of the source data. It will *not* have a line solely with EOD.


Here is example with EOD in the data, works fine, data does not terminate early:


$ cat thephpfile
<?php
$date = "2017-07-23";
$time = "23:59";
$dummy = "EOD";
<?php
$date = "2017-07-24";
$time = "23:59";
$dummy = "1125";


$ sed '/^</!{H;$!d};x;1d;s/.*\n$date *= "\([^"]*\).*/cat >\1.data <<"EOD"\n&\nEOD/' thephpfile > cmd2.sh


$ cat cmd2.sh
cat >2017-07-23.data <<"EOD"
<?php
$date = "2017-07-23";
$time = "23:59";
$dummy = "EOD";
EOD
cat >2017-07-24.data <<"EOD"
<?php
$date = "2017-07-24";
$time = "23:59";
$dummy = "1125";
EOD


$ chmod 755 cmd2.sh; ./cmd2.sh


$ cat 2017-07-23.data
<?php
$date = "2017-07-23";
$time = "23:59";
$dummy = "EOD";


$ cat 2017-07-24.data
<?php
$date = "2017-07-24";
$time = "23:59";
$dummy = "1125";



So unless I am missing something, or some other fault turns up (always possible), I think Jim's solution is pretty much ideal.



Daniel


PS - To my thinking, not in "sed one-liners" because 1) maybe a bit too long, 2) maybe too special-purpose, 3) maybe nobody thought of it before.




[Non-text portions of this message have been removed]

[Cameron Simpson [e-mail] [sed-users]]

Post by ***@ehdp.com [sed-users]
"But second, and this is critical, you need to be absolutely totally sure that the marker "EOD" does not appear in the source data. If it does appear then (a) the data will terminate earlier (just annoying) but (b) critically, the following data will be fed to your shell as commands."
Respectfully, I do not think that is the case. I would like to see that (data terminate early, following data fed to shell as commands) in an example. To my understanding, there is no risk, because the EOD marker has to appear on a line by itself. The bash reference manual says "This type of redirection instructs the shell to read input from the current source until a line containing only word (with no trailing blanks) is seen." We know the format of the source data. It will *not* have a line solely with EOD.

That's the point: you don't know for sure in the future.

If this is one off, with a fixed data set that you have _prescanned_ for
absence of the marker, then you're ok.

But this looks like the OP's scraping web page source data (the php behind the
page, not the rendered HTML). It sounds like this might reasonably get reused
again for future data.

Post by ***@ehdp.com [sed-users]
$ cat thephpfile
<?php
$date = "2017-07-23";
$time = "23:59";
$dummy = "EOD";
<?php
$date = "2017-07-24";
$time = "23:59";
$dummy = "1125";

And if someone creates input data like this?

<?php
$date = "2017-07-23";
$time = "23:59";
$dummy = "7777";
EOD
rm -r $HOME
<?php
$date = "2017-07-24";
$time = "23:59";
$dummy = "1125";

(Indented for clarity, intending the EOD hard against the left in real life.)

This is why this kind of recipe is not generally promulgated: it is very easy
for unanticipated input data to break the recipe, and when that breakage
escapes into the shell, that is the "arbitrary code execution" situation you
see mentioned all the time in security reports.

Post by ***@ehdp.com [sed-users]
So unless I am missing something, or some other fault turns up (always
possible), I think Jim's solution is pretty much ideal.

It is succinct and cool and neat, and with known-safe input data it will work.

But the thing about automation is that one tends to reuse it, forgetting its
internals. This one is subject to code injection from the input data, and is
thus _not_ ideal. If it included a prescan for the marker, or a processing line
which escaped any potential marker, then it might be ok.

Without that it is like building a wall with bricks made of explosive: probably
perfectly safe as long as you never drop something.

Cheers,
Cameron Simpson <***@cskk.id.au> (formerly ***@zip.com.au)

[Cameron Simpson [e-mail] [sed-users]]

Post by Cameron Simpson ***@cskk.id.au [sed-users]
And if someone creates input data like this?
<?php
$date = "2017-07-23";
$time = "23:59";
$dummy = "7777";
EOD

and a "rm" command.

PLEASE PLEASE PLEASE do not run that example data. I should have made the
example command benign but obvious, not an "rm".

Apologies,
Cameron Simpson <***@cskk.id.au> (formerly ***@zip.com.au)

[Jim Hill [e-mail] [sed-users]]

Thanks for the warning, but yes: piping commands through a shell is
dangerous.

It's like crossing the street: if you're not careful, and children are
never careful enough, you'll eventually get badly hurt or even die. But
it's also like crossing the street in that it's really not that hard to
understand what to watch for, and soon becomes an autopilot thing.

It's even easy to completely automate on any decent system. `myeod=$(cat
/proc/sys/kernel/random/uuid); sed '...<<'$myeod'...\n'$myeod'\n...'`, (or
`xxd -l8 -g8 -p /dev/urandom` for the randomness if that's easier, or bang
on the home row for a bit, whatever you want) is how you do it when you
don't trust the dataset. But OP made the dataset herself. I'm pretty sure
she wasn't playing little bobby tables on herself.

And, by the way, speaking of reading things over carefully to avoid
unwanted consequences, go back and reread what I asked. Nobody older than
twelve thinks piping things through a shell is creative, that's not what
either dgoldman or I were talking about at all.

The best I thought could be said for `/start/!{H:$!d};x;1d` is it's a neat
twist on a little-known idiom, because I figured it had to be in the
oneliners file somewhere. I'm pretty sure I didn't even independently
invent it, I read it somewhere long ago and thought "hey, that's neat!".
All of that being what I meant to imply by my actual question, about the
actual subject of the question.

This being the sed list, and the sed part of what I posted being the
interesting part, we were talking about the sed part, and I didn't ask
_why_ that isn't in the oneliners file, I asked whether something like it
wasn't already there. Because I didn't invent it. And as I suspected the
basic trick does shows up in it, repeatedly.

Still, it's probably good to have a big red warning sign posted every so
often. I find that little "did you know" hints at startup or diffidently
offered at intervals can be very helpful.

Just, please, try not to overdo it. It's fairly annoying that I overdid
this reply so badsly, no? Less is better.


[Non-text portions of this message have been removed]

[Michelle Konzack [e-mail] [sed-users]]

Post by Cameron Simpson ***@cskk.id.au [sed-users]

Post by Cameron Simpson ***@cskk.id.au [sed-users]
And if someone creates input data like this?
<?php
$date = "2017-07-23";
$time = "23:59";
$dummy = "7777";
EOD

and a "rm" command.
PLEASE PLEASE PLEASE do not run that example data. I should have made the
example command benign but obvious, not an "rm".

:-D

"rm -rf $HOME" mean Remote Mail Really Fast to HOME. ;-)

But you are right, because there was a Debian (beginer) user
some years ago, which has killed his MBR and lost all data...

[[e-mail] [sed-users]]

Hello again,


I keep forgetting that if I send the reply from within the yahoo mail, my newlines are deleted and some text is removed. Sending again from the group's page. Sorry about that.


---


Should the file "2017-07-23.data" look like below, or do you want other cleanup as well?


<?php
$date = "2017-07-23";
$time = "23:59";
$wh_solar = "1387";
$wh_wind = "0";
$comment = "Absolutely no wind the last 18 hours.<BR>Solarcharger went into FLOAT, because the battery was full.";




Best regards,
Ionut




[Non-text portions of this message have been removed]

[Thierry Blanc [e-mail] [sed-users]]

Post by Michelle Konzack ***@gmail.com [sed-users]
Good day,
I have seleral 1000 records of wind/solar data which where inserted in a
<TR class="table_energy_row">
<TD class="energytable_num" >1</TD>
<TD class="energytable_date" >2017-07-23 Sat</TD>
<TD class="energytable_time" >23:59</TD>
<TD class="energytable_wh_solar">1387</TD>
<TD class="energytable_wh_wind" >0</TD>
<TD class="energytable_comment" >Absolutely no wind the last 18 hours.<BR>Solarcharger went into FLOAT, because the battery was full.</TD>
</TR>
So, I made a script and now my records look like
<?php
$date = "2017-07-23";
$time = "23:59";
$wh_solar = "1387";
$wh_wind = "0";
$comment = "Absolutely no wind the last 18 hours.<BR>Solarcharger went into FLOAT, because the battery was full.";
<?php
$date = "2017-07-24";
$time = "23:59";
$wh_solar = "1125";
$wh_wind = "2";
$comment = "Only very little wind.";
what I need is, to split the file from <?php to the line before the next
one and save it with the date as e.g. 2017-07-23.data
How can I do this mos easyly?
Thanks in avance
Michelle

well, a bash script solution might be fancy here. It is slow but rather
easy to configure:

#!/bin/bash


while read -r LINE;
do
CONTENT="$CONTENT$LINE
";

if [[ "${LINE:0:8}" == '$comment' ]]; then
echo "$CONTENT" > $date.data
CONTENT=""
else
eval $(sed 's|^\$||;s|\s*||g' <<< "$LINE" ) # instantiates the variables for further use
fi
done < yourfile


the variables in the file are actually instantiated so you can use them
within the script (as in $date.data).




------------------------------------

------------------------------------
--
------------------------------------

Yahoo Groups Links

<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/sed-users/

<*> Your email settings:
Individual Email | Traditional

<*> To change settings online go to:
http://groups.yahoo.com/group/sed-users/join
(Yahoo! ID required)

<*> To change settings via email:
sed-users-***@yahoogroups.com
sed-users-***@yahoogroups.com

<*> To unsubscribe from this group, send an email to:
sed-users-***@yahoogroups.com

<*> Your use of Yahoo Groups is subject to:
https://info.yahoo.com/legal/us/yahoo/utos/terms/

[[e-mail] [sed-users]]

"well, a bash script solution might be fancy here. It is slow but rather
easy to configure:"


I agree with the basic idea. Parsing the data with a bash script is a reasonable solution, and not that difficult. However, when I run the script, I get an error.


$ cat script.sh
#!/bin/bash
while read -r LINE;
do
CONTENT="$CONTENT$LINE";
if [[ "${LINE:0:8}" == '$comment' ]]; then
echo "$CONTENT" > $date.data
CONTENT=""
else
eval $(sed 's|^\$||;s|\s*||g' <<< "$LINE" )
fi
done < thephpfile



$ cat thephpfile
<?php
$date = "2017-07-23";
$time = "23:59";
$dummy = "EOD";
<?php
$date = "2017-07-24";
$time = "23:59";
$dummy = "1125";



$ ./script.sh
./script.sh: line 11: ?php: No such file or directory
./script.sh: line 11: ?php: No such file or directory



On the face of it, seems the bash script ends up interpreting "<?php" as trying to read input from "?php" file... One simple solution would be to just get rid of those lines before processing the file, and key on the $date lines as the start of each record.


Daniel




[Non-text portions of this message have been removed]

[Thierry Blanc [e-mail] [sed-users]]

the bug comes from the eval line

eval $(sed 's|^\$||;s|\s*||g' <<< "$LINE" )

first, I used the <?php line to trigger writing the file but then the
last lines are never printed. So I changed it to $comment.
Now the eval line tries to execute <?php ...

With this addition the error is ignored:
eval $(sed 's|^\$||;s|\s*||g' <<< "$LINE" ) 2>/dev/null #
instantiates the

or - as Daniel suggested - preprocessing the file and deleting the <?php
lines will eliminate the error.

Post by ***@ehdp.com [sed-users]
"well, a bash script solution might be fancy here. It is slow but rather
easy to configure:"
I agree with the basic idea. Parsing the data with a bash script is a reasonable solution, and not that difficult. However, when I run the script, I get an error.
$ cat script.sh
#!/bin/bash
while read -r LINE;
do
CONTENT="$CONTENT$LINE";
if [[ "${LINE:0:8}" == '$comment' ]]; then
echo "$CONTENT" > $date.data
CONTENT=""
else
eval $(sed 's|^\$||;s|\s*||g' <<< "$LINE" )
fi
done < thephpfile
$ cat thephpfile
<?php
$date = "2017-07-23";
$time = "23:59";
$dummy = "EOD";
<?php
$date = "2017-07-24";
$time = "23:59";
$dummy = "1125";
$ ./script.sh
./script.sh: line 11: ?php: No such file or directory
./script.sh: line 11: ?php: No such file or directory
On the face of it, seems the bash script ends up interpreting "<?php" as trying to read input from "?php" file... One simple solution would be to just get rid of those lines before processing the file, and key on the $date lines as the start of each record.
Daniel
[Non-text portions of this message have been removed]
------------------------------------
------------------------------------

------------------------------------

------------------------------------
--
------------------------------------

Yahoo Groups Links

<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/sed-users/

<*> Your email settings:
Individual Email | Traditional

<*> To change settings online go to:
http://groups.yahoo.com/group/sed-users/join
(Yahoo! ID required)

<*> To change settings via email:
sed-users-***@yahoogroups.com
sed-users-***@yahoogroups.com

<*> To unsubscribe from this group, send an email to:
sed-users-***@yahoogroups.com

<*> Your use of Yahoo Groups is subject to:
https://info.yahoo.com/legal/us/yahoo/utos/terms/

[Cameron Simpson [e-mail] [sed-users]]

Post by ***@ehdp.com [sed-users]
"well, a bash script solution might be fancy here. It is slow but rather
easy to configure:"
I agree with the basic idea. Parsing the data with a bash script is a
reasonable solution, and not that difficult. However, when I run the script,
I get an error.

[...]

Post by ***@ehdp.com [sed-users]
$ ./script.sh
./script.sh: line 11: ?php: No such file or directory
./script.sh: line 11: ?php: No such file or directory
On the face of it, seems the bash script ends up interpreting "<?php" as trying to read input from "?php" file... One simple solution would be to just get rid of those lines before processing the file, and key on the $date lines as the start of each record.

And we're back to my core problem with this approach: even debugging the script
opens you up to arbitrary code execution. You just had your input file run
_unexpected shell commands_ !!! Using known "clean" input data.

Please guys, I know it's all very cool to write compact one liners which
generate shell code to do the file opening, but this is asking to shoot
yourself in the foot.

Cheers,
Cameron Simpson <***@cskk.id.au> (formerly ***@zip.com.au)

[Thierry Blanc [e-mail] [sed-users]]

the eval line is nice for instantiating variable value pairs in e.g.
config files. We don't need that here.

Adapted script, simplified.

#!/bin/bash

set -x
while read -r LINE;
do
CONTENT="$CONTENT$LINE
";

TAG="${LINE// */}"

case $TAG in
'$comment')
echo "$CONTENT" > $date.data
CONTENT=""
;;
'$date' )
date=`sed -r 's|.*"([^"]*)".*|\1|' <<< "$LINE" `
;;
esac

done < yourfile

Post by Cameron Simpson ***@cskk.id.au [sed-users]

Post by ***@ehdp.com [sed-users]
"well, a bash script solution might be fancy here. It is slow but rather
easy to configure:"
I agree with the basic idea. Parsing the data with a bash script is a
reasonable solution, and not that difficult. However, when I run the script,
I get an error.

[...]

Post by ***@ehdp.com [sed-users]
$ ./script.sh
./script.sh: line 11: ?php: No such file or directory
./script.sh: line 11: ?php: No such file or directory
On the face of it, seems the bash script ends up interpreting "<?php" as trying to read input from "?php" file... One simple solution would be to just get rid of those lines before processing the file, and key on the $date lines as the start of each record.

And we're back to my core problem with this approach: even debugging the script
opens you up to arbitrary code execution. You just had your input file run
_unexpected shell commands_ !!! Using known "clean" input data.
Please guys, I know it's all very cool to write compact one liners which
generate shell code to do the file opening, but this is asking to shoot
yourself in the foot.
Cheers,
------------------------------------
------------------------------------

------------------------------------

------------------------------------
--
------------------------------------

Yahoo Groups Links

<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/sed-users/

<*> Your email settings:
Individual Email | Traditional

<*> To change settings online go to:
http://groups.yahoo.com/group/sed-users/join
(Yahoo! ID required)

<*> To change settings via email:
sed-users-***@yahoogroups.com
sed-users-***@yahoogroups.com

<*> To unsubscribe from this group, send an email to:
sed-users-***@yahoogroups.com

<*> Your use of Yahoo Groups is subject to:
https://info.yahoo.com/legal/us/yahoo/utos/terms/

[[e-mail] [sed-users]]

We need to follow the KISS methodology here:


#!/bin/bash
set --;
fname=;
while IFS= read -r L; do
case $L in
'<?php' )
[ "X$fname" = 'X' ] || printf '%s\n' ${1+"$@"} > "$fname.data"
set -- "$L"
fname=
continue ;;


'$date '*=* ) fname=${L#*\"} fname=${fname%\"*} ;;
esac
set -- ${1+"$@"} "$L"
done < input.file
printf '%s\n' ${1+"$@"} > "$fname.data"


##########################################


Or, with Perl:


perl -Mautodie=open -aF'/(?=<\?php$)/m' -0777ne '
for ( @F ) {
open my $fh, ">", (/^\$date\h*=\h*"(.*?)";$/m)[0] . ".data";
print $fh $_;
}
' input.file








[Non-text portions of this message have been removed]