Non-affiliated buses for sale - thread

[Njsurf73]

Originally Posted by Robin97396

ECCB, if you remember what was the name of that web crawling search engine we were playing with last year that was finding buses so well?

It seemed to catch private sales, craigslist, Public Surplus and more, but not GovDeals or something like that. That sure narrows down the search process as you can limit it to your area.

If you scan back a few posts, Ourmefa seems to have a crawler that brings up some pretty cool results

Sent from my XT1254 using Tapatalk

[Robin97396]

Thanks Njsurf73, those listings from Ourmefa are what made me think of the webcrawler. I see the listings but not the name of the webcrawler. Somebody rub my nose in it!

[Northwest of Normal]

(Hey, newbie here; hope this post is OK. Quoting craigslist…) "1989 crown school bus for sale runs good ,new tires turbo detroit diesel 5 speed stick transmission $8000 call for an appointment. also i have a 1979 crown bus with a detroit diesel engine 5 speed transmission last inframe was in 2009 good tires. runs good. price $2000"

https://losangeles.craigslist.org/la...007031558.html

[Northwest of Normal]

https://images.craigslist.org/00k0k_...e9_600x450.jpg
https://images.craigslist.org/00D0D_...xm_600x450.jpg

[Njsurf73]

Originally Posted by Robin97396

Thanks Njsurf73, those listings from Ourmefa are what made me think of the webcrawler. I see the listings but not the name of the webcrawler. Somebody rub my nose in it!

I think it may have been his own crawler.

Sent from my XT1254 using Tapatalk

[cadillackid]

its not hard to build a crawler box.. there's numerous tutorials on the web for building them..

or you can write your own, crawl deeper, ignore robots.txt files, etc..

scrapy is a good start.. you can also find pythoin examples.. biggest part is parsing the returns you get..

ie you can search govdeals and such and look for certain keywords and then stores those URL's for future use, etc.. ie look for 'DT466', and pick off any results htat have it.. and save the direct URL's to those pages..

I dont kn ow what ourmefa is using.. or if he wrote his from scratch.
-Christopher

[Njsurf73]

Originally Posted by cadillackid

its not hard to build a crawler box.. there's numerous tutorials on the web for building them..

or you can write your own, crawl deeper, ignore robots.txt files, etc..

scrapy is a good start.. you can also find pythoin examples.. biggest part is parsing the returns you get..

ie you can search govdeals and such and look for certain keywords and then stores those URL's for future use, etc.. ie look for 'DT466', and pick off any results htat have it.. and save the direct URL's to those pages..

I dont kn ow what ourmefa is using.. or if he wrote his from scratch.
-Christopher

Me either, but I like his results! Lol
For those of us that aren't as tech savvy , we appreciate you tech guys.

Sent from my XT1254 using Tapatalk

[ourmefa]

I have been a software engineer for about 15 years now with a specific focus on the open source software Drupal. This software suite runs on a server using PHP for the scripting language (the same as what powers this forum).

Drupal has many similarities to software such as Wordpress, Joomla or WebGui, but it differs in that it is also considered a web framework. I have often explained Drupal to others by comparing Wordpress as being a hammer and Drupal being an entire garage full of tools. There are many many things you can do with Drupal without knowing a single line of coding. It is mostly following instructions and then pointing and clicking!!

For the bus auction crawler, I am using a module recipe that uses something called xpath to crawl the code on a page, and then using special selectors, pick out the important information from the page. Not just the URL of the page, but also things like updating the price of a particular listing each time it crawls.

I can also tell the system to ignore items that were picked up in a crawl but are not relevant. All of the data from each crawl is also stored in a database. This would allow me to gather things like VIN numbers and then cross reference them against previous auctions or other sites where a bus might be for sale.

For example: A lot of buses go up on govdeals and then 3 months later, the same vin numbers come up at either a dealer site or maybe on craigslist. I can then track that information back to the original auction and see how the item was listed at that time. This system would also allow others, who have skoolies, to claim their VIN number and fill in missing information that may not have been included in the other listings. Or maybe share their skoolie story. Not sure yet on that part.

I am just now getting the basics of it written, but once completed, I hope to expose for everyone to use here on Skoolie.net. I am just not sure how soon that will be lol. We are about 30 days from me having a lot of free time, so hopefully it will be soon!

Hope this helps everyone. Feel free to post suggestions for specific search criteria or information that you feel it would be good to have gathered in this kind of database.

[Robin97396]

Yup, that cleared it up alright. Tech has passed me by. I noticed in searches that there was a lot of information on building my own crawler or information scraper.

You make some very interesting finds. I'm not looking to buy anything. I do enjoy seeing the rare finds.

Thanks.

[ourmefa]

Originally Posted by Robin97396

Yup, that cleared it up alright. Tech has passed me by. I noticed in searches that there was a lot of information on building my own crawler or information scraper.

You make some very interesting finds. I'm not looking to buy anything. I do enjoy seeing the rare finds.

Thanks.

Well, one of my challenges has been to develop a lexicon of terms to search with. I can use everyone's help on this, techy or not!

I have gathered some of the more common terms such as manufacturers, skoolie/bus conversion lingo, engines, transmissions, rear end ratios, tire sizes, features/brands (retarder, air brakes, air ride, Onan, etc).

In the Drupal world this would be called a taxonomy. I will need help soon in defining as many of these as possible. This will allow me to use some ranking logic to filter the important items to the top of the result returns.

Many here would be very surprised by the sheer amount of data that can be gathered in one day of crawling just one or two auction sites, let alone ALL of Craigslist! LOL

My first pass at crawling just the bus category on govdeals generated about 2 gigs in the database. Needless to say it had to be paired down! Now I have it down to about 1/100th of that with much better data segmentation.

I am going to try and have this up and working for everyone in the next 30-45 days. I will probably poll a few of you to test it in the next couple of weeks.

Sent from my SM-G935T using Tapatalk

[Robin97396]

That sounds very cool. Thanks for your efforts.

We can certainly contribute to your taxonomy. We gots words.

[dan-fox]

Originally Posted by ourmefa

I have been a software engineer for about 15 years now with a specific focus on the open source software Drupal. <snip>

Drupal has many similarities to <snip>

For the bus auction crawler, I am using a module recipe that uses something called xpath to <snip>

I can also tell the system to <snip>. This would allow me to gather things like VIN numbers and then cross reference them against previous auctions or other sites where a bus might be for sale.

For example: A lot of buses go up on govdeals and then 3 months later, the same vin numbers come up at either a dealer site or maybe on craigslist. I can then track that information back to the original auction and see how the item was listed at that time. This system would also allow others, who have skoolies, to claim their VIN number and fill in missing information that may not have been included in the other listings. Or maybe share their skoolie story. Not sure yet on that part.

<snip>

Sounds very interesting, and useful! I can see tracking the sale price, VIN, and userid on the public auction sites so we can eventually make educated guesses as to what userid goes with what reseller, and their markup policies. It would be a powerful tool to be able to go into a negotiation knowing how much the seller has in it.

Don't worry about the gigs of data; we can always put it up on AWS or similar.

I have 35 yr software dev experience. I've been doing web back end stuff in Java for the last 15 or so. I don't know from Drupal but I have some Locomotive and more Documentum than I care to admit. Also I speak Xpath. Gimme a shout if I can be of any assistance.

We'll stop speaking Geek now, and return you to your regularly scheduled entertainment.

[cadillackid]

now thats bad ass!! love it.. and yeah I get it.. Drupal is an incredible tool box.. I havent done a lot of it, as I dont do a lot of web work... one of my guys does most of my web work.. I do a TON of back-end support code though..

PHP included., ive written a lot of PHP applications.. web and CLI... (did you know a source-guardian compiled PHP script is more efficient than a GCC-C++ compiled program?)

I love it when tech can meet old school like you are doing to help us track a bus through its various cycles after school-life!!

super stuff!!
-Christopher

[oricha1984]

So PublicSurplus called me today. The call was to remind me about a specific bus auction ending tomorrow.
I had no idea they did that . . .

Sent from my BLU LIFE ONE X using Tapatalk

[Njsurf73]

Wow. That's customer service.

Sent from my XT1254 using Tapatalk

[EastCoastCB]

Originally Posted by oricha1984

So PublicSurplus called me today. The call was to remind me about a specific bus auction ending tomorrow.
I had no idea they did that . . .

Sent from my BLU LIFE ONE X using Tapatalk

Yup they're really friendly.

[cadillackid]

did you buy the bus?
-Christopher

[EastCoastCB]

Originally Posted by oricha1984

So PublicSurplus called me today. The call was to remind me about a specific bus auction ending tomorrow.
I had no idea they did that . . .

Sent from my BLU LIFE ONE X using Tapatalk

Imagine how Roxy must feel, EVERY bus I bid on- they call HER number!

[oricha1984]

No bus for me yet, I'll hold off for a while longer

Sent from my BLU LIFE ONE X using Tapatalk

[PNW_Steve]

Originally Posted by EastCoastCB

Yup they're really friendly.

I wish GovDeals was that friendly.

When I bought my Bluebird I made pickup arrangements with the seller that GovDeals did not like. They banned me. Now if I want something on GovDeals my wife has to do the bidding.