Python Coders...Msaada

[Python Newbie]

Kwa nini hizi codes zinarudisha 'None'. Nimescrape kupata price ya hiyo item S%P 500 Index

import requests
url="https://www.bloomberg.com/quote/SPX:IND"
biz_r=requests.get(url)
from bs4 import BeautifulSoup
biz_soup=BeautifulSoup(biz_r.text,'html.parser')
print(biz_soup.find_all('span', attrs={'class': 'priceText__1853e8a5'}))

 

[Python Newbie]

Hili Jukwaa halina coders siku hizi..@MaxMase Stefano Mtangoo Graph Arduino Sentinel

 

[dronedrake]

Hili Jukwaa halina coders siku hizi..@MaxMase Stefano Mtangoo Graph Arduino Sentinel

most likely unakua detected as 'bot'

tafuta tutorial za ku ping kwa Selenium au jarib

import requests

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}

url="https://www.bloomberg.com/quote/SPX:IND"
biz_r=requests.get(url,headers =headers)
from bs4 import BeautifulSoup
biz_soup=BeautifulSoup(biz_r.text,'html.parser')
print(biz_soup.find_all('span', attrs={'class': 'priceText__1853e8a5'}))

 

[Python Newbie]

Thanks...though nayo imekataa...

 

[dronedrake]

Thanks...though nayo imekataa...

cheki na Selenium inamimick 'web browser' kwenye script

 

[Python Newbie]

cheki na Selenium inamimick 'web browser' kwenye script

Thanks...

 

[Python Newbie]

most likely unakua detected as 'bot'

tafuta tutorial za ku ping kwa Selenium.....

Sure...kwa code zinazofanana nimeweza kuscrape zoomtanzania

import requests
url="https://www.zoomtanzania.com/medicine-pharmaceutical-jobs"
zoom_r=requests.get(url)
zoom_r.text
from bs4 import BeautifulSoup
zoom_soup=BeautifulSoup(zoom_r.text,'html.parser')
title_zoom=zoom_soup.findAll('div', attrs={'class':'listing-card__header__title'})
print (title_zoom)

 

[dronedrake]

Sure...kwa code zinazofanana nimeweza kuscrape zoomtanzania

njia nyepesi ya ku detect bot kwenye web yako ni kutumia

onmousemove()

kwamba kwa human wa kawaida akiingia kwenye iyo site lazima mouse ita move japo kidogo, ila kwa scraping script, hakuna functon za kumuvuzisha iyo mouse, ndio maana unakua detected as a 'bot'

zoom hawajaweka anti scraping scripts kwenye page zao

 

[Freak]

Kwa nini hizi codes zinarudisha 'None'. Nimescrape kupata price ya hiyo item S%P 500 Index

import requests
url="https://www.bloomberg.com/quote/SPX:IND"
biz_r=requests.get(url)
from bs4 import BeautifulSoup
biz_soup=BeautifulSoup(biz_r.text,'html.parser')
print(biz_soup.find_all('span', attrs={'class': 'priceText__1853e8a5'}))

The code ipo Poa ni kwamba the CAPCHA verification Ndio inazuiia baadae nitatafuta Jinsi ya kudecode (nakumbuka nilona sehemu) but kwa nilivyo ona the best solution ni kutumia API yao japokuwa ina itaji build tools

Full disclosure I will try kutafuta njia through the sitemap na pia a way throuh the Capcha na api (nadhan ninayo biuld toolls installed)

 

[Python Newbie]

njia nyepesi ya ku detect bot kwenye web yako ni kutumia

onmousemove()

kwamba kwa human wa kawaida akiingia kwenye iyo site lazima mouse ita move japo kidogo, ila kwa scraping script, hakuna functon za kumuvuzisha iyo mouse, ndio maana unakua detected as a 'bot'

zoom hawajaweka anti scraping scripts kwenye page zao

Asante kwa shule...

 

[Python Newbie]

The code ipo Poa ni kwamba the CAPCHA verification Ndio inazuiia baadae nitatafuta Jinsi ya kudecode (nakumbuka nilona sehemu) but kwa nilivyo ona the best solution ni kutumia API yao japokuwa ina itaji build tools

Full disclosure I will try kutafuta njia through the sitemap na pia a way throuh the Capcha na api (nadhan ninayo biuld toolls installed)

Cool...ukija na solution uje utupe shule hapa

 

[Freak]

Cool...ukija na solution uje utupe shule hapa

Sitemap, selenium, na capcha zote zimekataa but nimepata an alternative za ku retrive data mpaka nipate njia nyingine ipi ni kutumia Alphavantage

 

[Python Newbie]

Sitemap, selenium, na capcha zote zimekataa but nimepata an alternative za ku retrive data mpaka nipate njia nyingine ipi ni kutumia Alphavantage

Thanks kiongozi...

 

[Mtangoo]

Hili Jukwaa halina coders siku hizi..@MaxMase Stefano Mtangoo Graph Arduino Sentinel

Kwa sababu ya kukosa u serious tumehamia Slack (cheki kwenye signature yangu)

 

[Mtangoo]

Sure...kwa code zinazofanana nimeweza kuscrape zoomtanzania

Nadhani Bloomberg wamezuia makusudi kupunguza traffic ya scrappers kwa sababu ya aina ya data walizonazo zina interest ya data scientists wengi

 

[Python Newbie]

Nadhani Bloomberg wamezuia makusudi kupunguza traffic ya scrappers kwa sababu ya aina ya data walizonazo zina interest ya data scientists wengi

Sure

 

[Python Newbie]

Kwa sababu ya kukosa u serious tumehamia Slack (cheki kwenye signature yangu)

Thanks...But hii link haifanyi kazi

This invite link is no longer active.

 

[Graph]

Ni vizuri uwe na tool inaitwa Postman kama unapenda kuscrape, itakurahisishia sana kumake http requests.

Haya kuhusu swala lako, hiyo site nimejaribu kwenye postman naona tatizo wanahisi wewe ni bot kama jamaa juu hapo alivyosema so wanakupush kwenye captcha page. Kuweza kuvuka hiyo ukiweka "User-Agent" kwenye headers kwanza inagoma coz wanataka more information user-agent peke yake haitoshi. Nilizoweka zikakubali ni hizi

user-agent : Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36

origin : https://www.bloomberg.com

referer: https://www.bloomberg.com/quote/SPX:IND

cookie : <cookie from chrome browser>

Cookie hiyo unaipata kwenye chrome browser, visit hiyo site in the browser, right click nenda inspect, fungua network, refresh page then uclick the first link uangalie Headers walizotuma, kuna cookie pale, include hiyo kwenye header basi, watadetect kua you are human from there. Kazi kwisha problem solved.

Hiyo cookie hujui lini ita-expire, so iki-expire probably ukihitaji kuscrape itagoma, so tafuta jinsi ya kupata cookie jar na kurefresh ujue wanavyoigenerate. Au njia nyingine ndefu unaweza tumia libraries ambazo zinascrape kwa kujifanya browser inakua na implementations zote done for you.

 

[Mtangoo]

Thanks...But hii link haifanyi kazi

Slack

bit.ly

 

[Mtangoo]

Cookie hiyo unaipata kwenye chrome browser, visit hiyo site in the browser, right click nenda inspect, fungua network, refresh page then uclick the first link uangalie Headers walizotuma, kuna cookie pale, include hiyo kwenye header basi, watadetect kua you are human from there. Kazi kwisha problem solved.

Umeinasa kama ulimbo aisee!

Hiyo cookie hujui lini ita-expire, so iki-expire probably ukihitaji kuscrape itagoma, so tafuta jinsi ya kupata cookie jar na kurefresh ujue wanavyoigenerate. Au njia nyingine ndefu unaweza tumia libraries ambazo zinascrape kwa kujifanya browser inakua na implementations zote done for you.

Lazima kutakuwa na modules tu za kufanya hivyo. Sijaijaribu hii lakini maelezo yanaonesha inaweza kuwa msaada

GitHub - borisbabic/browser_cookie3: This is a fork of browser_cookie

This is a fork of browser_cookie. Contribute to borisbabic/browser_cookie3 development by creating an account on GitHub.

github.com