RE: Someone put another coin in the meter?

#46 | RE: Someone put another coin in the meter? datum

Mon May 02, 2022 10:56 pm

Zitat von Mr. 36654 im Beitrag #45
Zitat von zer0netgain im Beitrag #44
Zitat von Mr. 36654 im Beitrag #41
I get your message, but we are kinda dancing on the deck of the Titanic with the stern well above the water...

Quick, someone grab the Heart of the Ocean from Rose before she tosses it into the Atlantic....the silly moo.

Ya know, you just had to go there........

But, then again, Kate Winslet does have an attractive upper body

Then. Now everything is probably sagging.

I�m a Notary Public. RESPECT MY AUTHORITY!

reply

#47 | RE: Someone put another coin in the meter? datum

Mon May 02, 2022 11:25 pm

Zitat von zer0netgain im Beitrag #46
Zitat von Mr. 36654 im Beitrag #45
Zitat von zer0netgain im Beitrag #44
Zitat von Mr. 36654 im Beitrag #41
I get your message, but we are kinda dancing on the deck of the Titanic with the stern well above the water...

Quick, someone grab the Heart of the Ocean from Rose before she tosses it into the Atlantic....the silly moo.

Ya know, you just had to go there........

But, then again, Kate Winslet does have an attractive upper body

Then. Now everything is probably sagging.

I'm guessing ours is worse than hers

Old Forum Orphan

duccrazydave likes the post

reply

#48 | RE: Someone put another coin in the meter? datum

Tue May 03, 2022 12:16 pm

Zitat von Mr. 36654 im Beitrag #47
Zitat von zer0netgain im Beitrag #46
Zitat von Mr. 36654 im Beitrag #45
Zitat von zer0netgain im Beitrag #44
Zitat von Mr. 36654 im Beitrag #41
I get your message, but we are kinda dancing on the deck of the Titanic with the stern well above the water...

Quick, someone grab the Heart of the Ocean from Rose before she tosses it into the Atlantic....the silly moo.

Ya know, you just had to go there........

But, then again, Kate Winslet does have an attractive upper body

Then. Now everything is probably sagging.

I'm guessing ours is worse than hers

If my assets were hanging down at my knees, it would be an improvement.

I�m a Notary Public. RESPECT MY AUTHORITY!

reply

#49 | RE: Someone put another coin in the meter? datum

Tue May 03, 2022 1:03 pm (Last edited: Tue May 03, 2022 1:05 pm)

v8guy

Show info

Posts:	88
Registered since:	11.02.2019
Location:	London, UK

Here, an example of structured JSON output from my forum scraper:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
 

{
    "id": "t_4361",
    "title": "What fitment is this..??",
    "author": "Plowy",
    "forum": "https://www.r1200rsforum.com/f15-accessories.html",
    "forum_code": "f15",
    "posts": [
        [
            {
                "attachments": [
                    "https://files.homepagemodules.de/b797706/f15t4361p47901n2_ynIVDLOj.jpg"
                ],
                "author": "Plowy",
                "text": "Can you tell.me what top box this is for..??",
                "post_number": 1,
                "message_id": "47901",
                "timestamp": "2022-05-01 21:59:00",
                "epoch": 1651438740
            },
            {
                "attachments": [
                    "https://files.homepagemodules.de/b797706/f15t4361p47904n2_fxXeSYpo.jpg",
                    "https://files.homepagemodules.de/b797706/f15t4361p47904n4_kmxhYQBE.jpg"
                ],
                "author": "Lee",
                "text": "That's the OEM rear rack for the R and RS.It will accept the BMW 30 liter top box and the BMW Multi Vario top box.",
                "post_number": 2,
                "message_id": "47904",
                "timestamp": "2022-05-02 12:08:00",
                "epoch": 1651489680
            }
        ]
    ]
}
 
 

There are a few assumptions I've made, such as stripping out quoted text from posts, but this code now works quite reliably to turn the forum's content into a structured format. I still need to write it out to disk and grab all the attachments, but otherwise it's complete.

I have an idea for re- presenting all the extracted material once it's been pulled.. it shouldn't be too difficult to write some code to post each thread into another forum, using a bot account for the purpose. On the forum, the poster of each message would be the bot, and the timestamps wouldn't match the originals, but with the information pulled from the original posts such metadata could be presented in a readable way.

@grumpygoat - if you'd be willing to host the material we could look into creating an area for it on your forum and experimenting with re-submitting all of the posts there.

The biggest challenge might be sifting through the many hundreds of posts and determining which are worth archiving, but maybe that can be a community effort.

Current: 2015 BMW R1200RS Sport SE, 2003 Yamaha TDM900
Past: 1999 Honda CLR125, 1995 Honda CB500, 1998 Triumph Sprint Sport, 1995 Honda VFR750FS, 2006 Suzuki DRZ400SM, 2001 Triumph Speed Triple, 2015 Honda VFR800X

"Being shot out of a cannon will always be better than being squeezed out of a tube" - HST

these members like the post: Ali-bear, Freddyfruitbat, Mr. 36654, Ray and duccrazydave

reply

#50 | RE: Someone put another coin in the meter? datum

Tue May 03, 2022 1:10 pm

Also, which sub forums do we want to archive?

I am currently excluding the following:

F21 - off topic
F22 - other bikes
F3 - New member intros
F41 - buy and sell
f35 - guest area
f5 - forum operation
f58 - forum tutorials

Maybe better to create an include list and ignore everything else?

Current: 2015 BMW R1200RS Sport SE, 2003 Yamaha TDM900
Past: 1999 Honda CLR125, 1995 Honda CB500, 1998 Triumph Sprint Sport, 1995 Honda VFR750FS, 2006 Suzuki DRZ400SM, 2001 Triumph Speed Triple, 2015 Honda VFR800X

"Being shot out of a cannon will always be better than being squeezed out of a tube" - HST

reply

#51 | RE: Someone put another coin in the meter? datum

Tue May 03, 2022 1:28 pm

Buy and sell to me is a very important of all forums.

Lupin Blue R1200RS SE

reply

#52 | RE: Someone put another coin in the meter? datum

Tue May 03, 2022 1:30 pm

Being that I enjoy the pointless banter of this group, I would miss the content of "F21-Off Topic" and "F3-New Member Intros". However, I'm not the guy providing a new internet home........

Old Forum Orphan

reply

#53 | RE: Someone put another coin in the meter? datum

Tue May 03, 2022 1:51 pm

Zitat von Grumpy Goat im Beitrag #33
Zitat von zer0netgain im Beitrag #31
Zitat von Grumpy Goat im Beitrag #30
....takes me 2 flipping days just to get out of Texas going west.

How slow do you ride?

Heheh ... fast enough to cross the widest dimension of VA in 6 hours ... I exaggerated a bit about 2 days of course, but it would take 11 hours to go from my house to El Paso and we have decent speed limits here.

Driving I-10 across the southern US is a soul-crushing task, but seeing this sign when crossing the Eastern border of Texas out of Louisiana just feels cruel.

elpaso.jpg - Bild entfernt (keine Rechte)

these members like the post: Lee, Vtbob, Richard230, Duckbubbles, Grumpy Goat and duccrazydave

reply

#54 | RE: Someone put another coin in the meter? datum

Tue May 03, 2022 1:57 pm

Zitat von L2RKE im Beitrag #51
Buy and sell to me is a very important of all forums.

But do you care about having an archive of posts to it?

Current: 2015 BMW R1200RS Sport SE, 2003 Yamaha TDM900
Past: 1999 Honda CLR125, 1995 Honda CB500, 1998 Triumph Sprint Sport, 1995 Honda VFR750FS, 2006 Suzuki DRZ400SM, 2001 Triumph Speed Triple, 2015 Honda VFR800X

"Being shot out of a cannon will always be better than being squeezed out of a tube" - HST

reply

#55 | RE: Someone put another coin in the meter? datum

Tue May 03, 2022 2:03 pm

Zitat von v8guy im Beitrag #54
Zitat von L2RKE im Beitrag #51
Buy and sell to me is a very important of all forums.
But do you care about having an archive of posts to it?

Well exactly - I think if someone like you is prepared to invest time and effort into this (and other members too, as it looks like a lot of manual work will also be needed) we the users should be prepared to be even more circumspect about what to archive, otherwise even with the best will in the world, it's not going to happen. Maybe restrict it to the 'technology' subforums only? Ignore threads with only a small number of posts (less likely to be useful)?

KC100->CB100N->CB250RS--------->DL650AL2->R1200RS

reply

#56 | RE: Someone put another coin in the meter? datum

Tue May 03, 2022 2:48 pm

Zitat von TriangleRider im Beitrag #53

Heading west from the Alleghany Plateau, you desperately cling to any rolling topography you can find thru Michigan or Ontario, but eventually, the trees disappear and you're alone and exposed on the Plains. For miles. And miles. You wonder why you choose to do this.

Old Forum Orphan

these members like the post: Duckbubbles, TriangleRider and duccrazydave

reply

#57 | RE: Someone put another coin in the meter? datum

Tue May 03, 2022 3:09 pm

Zitat von v8guy im Beitrag #50
Also, which sub forums do we want to archive?

I am currently excluding the following:

F21 - off topic
F22 - other bikes
F3 - New member intros
F41 - buy and sell
f35 - guest area
f5 - forum operation
f58 - forum tutorials

Maybe better to create an include list and ignore everything else?

Well done v8guy. Your efforts are much appreciated.

Is it much more effort to just scrape the lot? At least we would then have it and likewise the bot could load it all to Grumpy's forum (with his permission of course).

OlieL likes the post

reply

#58 | RE: Someone put another coin in the meter? datum

Tue May 03, 2022 7:28 pm

Zitat von v8guy im Beitrag #49

There are a few assumptions I've made, such as stripping out quoted text from posts, but this code now works quite reliably to turn the forum's content into a structured format. I still need to write it out to disk and grab all the attachments, but otherwise it's complete.

I have an idea for re- presenting all the extracted material once it's been pulled.. it shouldn't be too difficult to write some code to post each thread into another forum, using a bot account for the purpose. On the forum, the poster of each message would be the bot, and the timestamps wouldn't match the originals, but with the information pulled from the original posts such metadata could be presented in a readable way.

Good to hear that you have made progress in this regard.

Zitat von v8guy im Beitrag #49

@grumpygoat - if you'd be willing to host the material we could look into creating an area for it on your forum and experimenting with re-submitting all of the posts there.

The biggest challenge might be sifting through the many hundreds of posts and determining which are worth archiving, but maybe that can be a community effort.

Therein lies the rub. I was there when the original forum died and this one was created, very shortly beating the RSWasserboxer.com forum to the punch. I tried to get community assistance to [manually] sift through the post and sift the wheat from the chaff, and the response was lacklustre.

I am not opposed to hosting material, but not a data dump. Only posts that have actual content that a person looking for information on the RS would want to invest the time to read, e.g. DIY articles, posts / articles about little known features and how to use them, posts / articles about problems / issues and how to fix them, etc. After all that was the impetus for the creation of RSWasserboxer.com and the Forum Community. So any new stuff would have to fit that vision.

Regards,
Grumpy Goat
RSWasserboxer.com
2016 BMW R1200RS
2014 BMW S1000R

these members like the post: Freddyfruitbat, darrell, Mark, Duckbubbles, Ray Bridgman and duccrazydave

reply

#59 | RE: Someone put another coin in the meter? datum

Tue May 03, 2022 7:28 pm

Zitat von Ray im Beitrag #57

Is it much more effort to just scrape the lot? At least we would then have it and likewise the bot could load it all to Grumpy's forum (with his permission of course).

There are in excess of 4000 threads on the site, it's a lot of scraping!
I've had to introduce a throttle to the bot as I believe the connection resets I get after a short period are a result of the forum barring access to my IP for a while, so better to be more selective, in my opinion.

Current: 2015 BMW R1200RS Sport SE, 2003 Yamaha TDM900
Past: 1999 Honda CLR125, 1995 Honda CB500, 1998 Triumph Sprint Sport, 1995 Honda VFR750FS, 2006 Suzuki DRZ400SM, 2001 Triumph Speed Triple, 2015 Honda VFR800X

"Being shot out of a cannon will always be better than being squeezed out of a tube" - HST

Ray likes the post

reply

#60 | RE: Someone put another coin in the meter? datum

Tue May 03, 2022 7:35 pm

Zitat von Grumpy Goat im Beitrag #58

I am not opposed to hosting material, but not a data dump. Only posts that have actual content that a person looking for information on the RS would want to invest the time to read, e.g. DIY articles, posts / articles about little known features and how to use them, posts / articles about problems / issues and how to fix them, etc. After all that was the impetus for the creation of RSWasserboxer.com and the Forum Community. So any new stuff would have to fit that vision.

I absolutely agree. It'll likely be relatively simple to make the bot crawl the extracted archive and create new posts with the material, but the curation of that archive will need to be done by a site administrator.. namely you, and also in conjunction with the community to determine what info is worth retaining and hosting. And there's the difficult bit!

Anyone got any better ideas on how to present and curate the archive?

Current: 2015 BMW R1200RS Sport SE, 2003 Yamaha TDM900
Past: 1999 Honda CLR125, 1995 Honda CB500, 1998 Triumph Sprint Sport, 1995 Honda VFR750FS, 2006 Suzuki DRZ400SM, 2001 Triumph Speed Triple, 2015 Honda VFR800X

"Being shot out of a cannon will always be better than being squeezed out of a tube" - HST

these members like the post: Freddyfruitbat and Ray

reply

RE: Someone put another coin in the meter? - 4