Please enable JavaScript.

Coggle requires JavaScript to display documents.

Python (modules: libraries hold functions need to be downloaded, not in…

- - - - sys.stdout.write("statement")
    - - inputStatement = sys.stdin.readline([number])
        read line from the keyboard;
        read first number of characters from the input
  - - - app: use the subtraction between time before running the function and the time after running the function to get the function's running time
    - - if not filling in tuple, it will show the time of localtime()
      - tuple=(year,month,day,hour,min,sec,day of the week, 6,0,0)
        mon-0,tue-1...
        day of the year - 0
        daylight savings time or not（1/0）: 夏时制
    - - ex: t = time.localtime()
        year = t[0]
  - - - app: crawling web - stock
        
        url="http://www.google.com"
        #User-Agent is added for avoiding forbidden Error,varies with browser:
        headers={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'}
        request = urllib.request.Request(url, headers = headers)
        htmlData = urllib.request.urlopen(request)
        #transfer html into readable text
        data = htmlData.read()
        #decode
        data1 = data.decode("utf-8")
        
        User-Agent of chrome:
        https://developers.whatismybrowser.com/useragents/explore/software_name/chrome/
      - app: crawling web - weather
  - - - Install: cmd > cd .../python/Scripts > "pip install beautifulsoup4" > install successfully
      - Prettify
        
        parse html
        
        from bs4 import BeautifulSoup
        #messy html code > parse it
        soup = BeautifulSoup("<html><p>aaa <strong>bbbb</html>", "html.parser")
        #prettify- completed html code, make html code neat
        soup.prettify()
        print(soup)
        
        get html tag content
        
        the first one: tag = soup.tagName. || tag = soup.find("tagName")
        e.g. soup.title/head/a... || soup.body.b
        get array contents in tag: tag.contents
        get direct children tags(直系) in the tag: array = tag.children
        get all children tags in the tag: array2 = tag.descendants
        
        all of them: soup.find_all()
        
        soup.find_all(id='idName' || href = re.compile("..."))
        e.g. soup.find_all(id = "link2")
        soup.find_all(href=re.compile("elsie"))
        But cannot find class = "..."
        
        soup.find_all("tageName" || re.compile("..."))
        e.g. array = soup.find_all('a')
        array = soup.find_all(re.compile("^b"))
        array[0]...
        
        soup.find_all("tagname", id='idName')
        e.g. soup.find_all("a",id = "link2")
        
        .string: if the tag doesn't have or have exactly one child tag without others, then return the content of this tag, otherwise return none.
        e.g. soup.strong.string > bbbb
        soup.p.string > none [because it contains aas except tag strong]
        
        .parent: the parent tag
        e.g.: soup.parent
        Notice: tag.string.parent - tag.string return the content of the tag, so its parent will return this tag rather than the parent of the tag.
        
        .find()
        
        find tag by class: e.g. soup.find('div',{'class':'className'})
    - - Install: cmd > cd .../python/Scripts > "pip install requests" > install successfully
      - crawling the website
        
        Get Response: data = requests.get("url")
        
        parse to html: soup = BeautifulSoup(data.text, "html.parser")
        
        find necessary tag: soup.find('div',{'class':'info'})
      - .content vs .text
        
        content中间存的是字节码，而text中存的是Beautifulsoup根据猜测的编码方式将content内容编码成字符串。直接输出content，会发现前面存在b'这样的标志，这是字节字符串的标志，而text是'，没有前面的b,对于纯ascii码，这两个可以说一模一样，对于其他的文字，需要正确编码才能正常显示。大部分情况建议使用.text，因为显示的是汉字，但有时会显示乱码，这时需要用.content.decode('utf-8')，中文常用utf-8和GBK，GB2312等。这样可以手工选择文字编码方式。所以简而言之，.text是现成的字符串，.content还要编码，但是.text不是所有时候显示都正常，这是就需要用.content进行手动编码。不明白可追问。
      - .get vs .post: return response obj- all info from the server;
        include 响应头，响应状态码.
        .content and .text can store the html part
  - - - create one graph:
        each time finishes the graph, show function needs to be invoked to display the graph
        ## Example: coordinates
        
        #plot graph: a square
        pt.plot([x1,x2,x3,x4],[y1,y2,y3,y4]) OR pt.plot(list1,list2,['linecolor',linewidth=number])
        pt.show() : after .plot(), the graph needs to be shown
        
        add labels in the graph
        
        pt.title("titleName")
        
        add labels for x/y axis
        
        pt.xlabel('xAxisName')
        
        pt.ylabel('yAxisName')
        
        change colors
        
        tick color: x,y轴上标记的小竖线
        
        pt.tick_params(axis="x/y", color="white")
        
        create one graph with multiple lines
        
        pt.plot(x1,y1,'color',linewidth=3.0)
        pt.plot(x2,y2,'color2',linewidth=4.0)
        
        # import matplotlib.pyplot as a simple name
        import matplotlib.pyplot as pt
      - create several graphs in one figure(canvas):
        #example: matplotlib>figure.py
        
        # step1: get the figure
        fig = pt.figure()
        
        #step2: get the patch/canvas
        rect = fig.patch
        # set the background color:
        rect.set_facecolor("green")
        
        #step3: paint graph 1: divide the canvas into nrow and ncol, then paint the graph in the index piece of canvas(from left to right, from top to bottom).
        graph1 = fig.add_subplot(nrow,ncol,index,graph background color)
        # set the coordinates and the line color/width
        graph1.plot(x,y,'red',linewidth=4.0)
        #show the graph:
        graph1.show()
        
        e.g.#将画布分割成3行4列，图像画在从左到右从上到下的第9块
        fig.add_subplot(349) OR (3.4.9)
        
        But if the index is greater than 10, it cannot be written without a comma.
        e.g. fig.add_subplot(3,4,10)
        
        # new version use facecolor while old us axisbg
        graph1 = fig.add_subplot(3,4,1,facecolor = "black")
        
        whole example: http://blog.csdn.net/zahuopuboss/article/details/54895890
        
        import matplotlib.pyplot as plt
        from numpy import *
        fig = plt.figure()
        ax = fig.add_subplot(2,1,1)
        ax.plot(x,y)
        ax = fig.add_subplot(2,2,3)
        ax.plot(x,y)
        plt.show()
        
        attributes of the graph
        
        color
        
        #spines color
        graph1.spines["top/left/right/bottom"].set_color('w')
        
        #tick color:
        graph1.tick_params(axis='x/y', color="white")
        
        # graph title
        graph1.set_tittle("title", color="white")
        
        #labels
        
        graph1.set_xlabel("x axis", color="white")
        
        graph1.set_ylabel("y axis", color="white")
    - - it can save the graph and check the details in it
      - wspace/hspace is for the figure when multiple graphs are too close with each other
    - - #Step1: import libs
        import matplotlib.pyplot as plt
        import numpy as np
      - #Step2: set the position of bars- array from 0-6; the gap between bars is 0.5 (the specification of the distance between the bars)
        pos = np.arange(6)+0.5
      - #Step3: create vertical of horizontal bar chart
        plt.bar(pos,(4,8,12,3,17,6),align='center', color='blue')
        OR plt.plot(kind='bar/barh')
        #horizontal bar chart
        plt.barh(pos,(4,8,12,3,17,6),align='center', color='blue')
        plt.show()
      - Labels
        
        # add title for it
        plt.title('titleName', color='white')
        #add label for x/y axis
        plt.xlabel('labelName',color='red')
        plt.ylabel('labelName', color='red')
        
        #add labels for each bar
        name = ['a','b','c','d','e','f']
        plt.xticks(pos,name)
        #for horizontal bar chart
        plt.yticks(pos,name)
      - color
        
        #tick color:
        plt.tick_params(axis='x/y', color="white")
        
        #spines color
        graph1.spines["top/left/right/bottom"].set_color('w')
      - #Adjust the margin between the chart and the canvas
        plt.subplots_adjust(left=.11, bottom=.12, right=.94)
    - - # Step1: import matplotlib
        import matplotlib.pyplot as plt
      - #Step2: draw the pie chart, both sizes and colors are an array. startangle can adjust the angle of the pie chart.
        plt.pie(sizes, clolors=colors, startangle = 90, shadow=True, labels = labels )
        
        #simple one:
        # the size of each part in the chart
        size = [1,2,3,4,5]
        # plot the chart: but it will be an oval
        plt.pie(size)
        
        # set color, size for the pie chart
        plt.pir(sizes, colors = colors, startangle = 90, labels = labels)
        #labels see the below bold labels
      - #step4: show the chart
        plt.show()
      - #step3: draw an circle by making the equal axis
        plt.axis("equal")
        # add legend : 饼图旁边的注释框 https://matplotlib.org/users/legend_guide.html
        # add labels
        labels = ['2011','2012','2013','2014','2015']
        # legend title, loc is the position of legend: best/left/center/right/lower/upper
        plt.legend(title='title', loc = "lower left")
    - - #Step1: get the figure
        fig = plt.figure()
        # create an sub plot as 3d
        chart = fig.add_subplot(1,1,1,projection = '3d')
      - #Step 2: set X,Y,Z
        X,Y,Z = [1,2,3,4],[2,3,4,5],[3,4,5,6]
        # plot the 3d graph
        chart.plot_wireframe(X.Y,Z) OR chart.plot(X.Y,Z)
        plt.show()
      - # plot scatter diagram/plot: 散点图https://www.cnblogs.com/shanlizi/p/6850318.html
        # 2d，c-color, s=point size, marker=point shape
        chart.scatter(x,y,c='b',s=10, marker='o',...)
        # 3d scatter diagram
        chart.scatter(x,y,z,c='b',s=10, marker='o',...)
        #Notice: multiple scatter plots in one canvas canbe draw by repeating the scatter function several times.
      - labels
        
        #set labels for axis: x/y/z
        chart.set_xlabels('x')
        chart.set_ylabels('y')
        chart.set_zlabels('z')
      - 3d Bar Graph
        
        #import numpy
        import numpy as np
        # Repeat Step1
        
        # set x,y,z,dx,dy,dz
        x = [1,2,3,4]
        y = [2,3,4,5]
        z = [0,0,0,0]
        dx = np.ones(4)
        dy = np.ones(4)
        dz = [1,2,3,4]
        
        # plot 3d bar
        chart.bar3d(x,y,z,dx,dy,dz, color='cyan')
        plt.draw()
      - axes3d
        
        # matplotlib tutorial : https://matplotlib.org/mpl_toolkits/mplot3d/api.html
        # Return a tuple X, Y, Z with a test data set.
        x,y,z = axes3d.get_test_data(0.05)
        # draw the picture https://matplotlib.org/mpl_toolkits/mplot3d/tutorial.html
        chart.plot_wireframe(x,y,z, rstride=10,cstride=10)
    - - #use 30 users' age column as the data to finish the histogram
        users.age.hist(bins=30)
      - # pandas data
        pData = users.groupby('a').size().order(ascending = False)[:50]
        # paint
        pData.order().plot(kind='barh', figsize=[10,15])
  - - - # dtype- data type, if none, infer the data
        np.arange([start,] stop[, step,], dtype=None)
        http://blog.csdn.net/qianwenhong/article/details/41414809
      - # returns an array filled with 1 with desired data type https://docs.scipy.org/doc/numpy/reference/generated/numpy.ones.html
        np.ones(number)
  - - - # set a series OR you can set index for the series
        s = pd.Series([10,"namaste",23.5,"aa"][,index=["a","b","c","d"]])
        # get each item in series
        s[0] OR s["a/b/c/d"]
      - # put dictionary into it
        d = {"a":11,"b":33}
        se = pd.Series(d)
        
        # get the item
        se["a"]
        #get specified items: b:33
        se[se >/== 11]
        #check if all items satisfy the condition: a false, b true
        se > 11
        # check if the item in se: true/false
        print("a" in se)
        
        #update one item value
        se["a"] = 20
        #update items meet the demand
        se[se >11] = 40
        # operate all items: all the number divide 10
        se/10
        
        #a false, b false
        se.isnull()
    - - # Step1: set the data
        #attention: the first character of the column must be capitalized.
        data={
        "Students":["a","b","c"],
        "Math":[99,87,100],
        "Sports":["badminton","basketball","TT"]
        }
      - #Step2: create dataFrame, columns-table columns in order,
        students = pd.DataFrame(data, columns=["Students","Math","Sports"])
        #print result
        print(students)
      - CSV related operations
        
        Read CSV file
        
        #Just read needed columns(first and sec) from the file
        newCSV = pd.read_csv("fileName.csv", usecols=[0,1], sep='\t', names=names, encoding="ISO-8859-1")
        # if has decoding problem, you should add the encoding attribute
        
        #Read CSV file
        pd.read_csv("csvFileName")
        # set new names for the first row， header=0 hide the original header.
        # Caution>> it would not change the file, just change the display of the data
        pd.read_csv("csvFileName", names=['a','b','c'], header=0)
        
        # get data from the web: http://files.grouplens.org/datasets/movielens/ml-100k/u.user
        # data is seperated by '|'
        #Caution: not define the column name with two word with a space, because users.Zip Code will cause syntax error.
        users = pd.read_csv("ml-100k/u.user", sep = "|", name=["User_ID", "Age","Gender","Occupation“,"Zip_Code"])
        
        #Slicing the data: get data from 10 to 14 lines [Start from 0]
        —— not related to user id
        users[10:15]
        # get all lines after 900 [include 900] OR before 3[not include 3]
        users[900:] OR users[:3]
        #Caution: negative number [give anything except the last two rows]
        users[:-2]
        
        # get and display the head data: 'number' lines from the beginning, default is 5
        users.head(number)
        # get the last 'numer' lines of data, default is 5
        users.tail(number)
        
        # get one specified column and rows
        users["Gender"].head(10) OR users.Gender
        # get several columns
        users[["Gender","Zip Code"]].head()
        
        #Boolean Expression
        users[users.Age > 25]
        users[(users.Age > 25) & (users.Gender == "M")].head()
        users[(users.Age > 25) | (users.Gender == "M")].head()
        # isin
        users[users.User_ID.isin(users2.index)]['columnName']
        
        Indexing and Other functions
        
        #return the data type of all columns
        users.dtypes
        
        #Calculate the mean/max/min/... for integer columns
        users.describe()
        
        #Indexing
        #temporarily change the index with id
        users.set_index("User_ID").head()
        #permanantly change the index with id
        users.set_index("User_ID", inplace = True)
        #reset index
        users.reset_index("User_ID", inplace = True)
        
        #get specified rows-put in an array
        users.ix[[1,12,44]]
        #Attention: if the index changes, the slicing will still start from 0, but the ix will choose according to the index number
        
        write CSV file
        
        #create a new CSV file
        newCSV.to_csv("newCsvFileName.csv")
      - Merge DataFrames
        
        #Step1: initialize two frames
        frame1 = pd.DataFrame({"key":range(4), "frame1":
        ["a","b","c","d"]})
        frame2 = pd.DataFrame({"key":range(2,6),"frame2":["q","v","w","x"]})
        
        #Step2: Merge when key equals: 3 columns-frame1,key.frame2
        pd.merge(frame1, frame2, on="key")
        # decide how to merge:
        right means that anything of frame2 will be remained while frame1 only contains common rows [not exist part will befilled with NaN]
        outer: remain all of the rows both in frame1&frame2, merge common rows
        inner: default, only remains common rows
        pd.merge(frame1, frame2, on="key", how="right/left/outer/inner")
        http://blog.csdn.net/zhouwenyuan1015/article/details/77334889
      - #concatenate[kən'kætɪneɪt] several frames- not merge
        pd.concat([frame1,frame2,...])
        #English
        http://pandas.pydata.org/pandas-docs/stable/merging.html
        #Chinese
        http://blog.csdn.net/stevenkwong/article/details/52528616
      - #group
        users2 = users.groupby("Gender")
        #count not null elements
        users2.count().head()
        # the total number of records
        users2.size().head()
        #order the data
        users2.order(ascending=False)
        # sort
        users2.sort([("name","id")] , ascending = False).head()
      - Heavy Data Manipulation
        
        #e.g.
        users.groupby("Name").size().order(ascending=False)[:20] OR users.Name.value_counts()[:20]
        
        # get three columns: name size mean
        stats = users.groupby("name".agg({'rating':[np.size,np.mean]}))
        #display head data
        stats.head()
        
        # generate a table: see the pandas web
        pivoted = users.pivot_table(index=['User_ID',"Gender"], columns = ['writer'], values = 'rating', fill_value=0)
        # add a new column in the pivot table
        pivoted['newCol'] = values
- - - - -- || / || * || %(1 mod 5 = 1)
    - - String: characters noted in quotation marks; Ex: random='abc def gh'
        
        get single characture: random[5]=d
        
        get substring: random[0:3] -> start from '0', end in 3; [0:]: start from '0' to the end of the string.
        
        str(obj): transfer object to string
        
        concatenation /kənkatəˈneɪʃn/ of strings;
        the action of linking things together in a series, or the condition of being linked in such a way
        
        str3 = str1 + str2
        
        string.upper() - capitalize the string
        
        array = string.split("character"): divides the string with character
      - List function
        
        use 2 square bracket: shoppingList=["egg","apple"]
        
        get specific item: array2[1]
        
        update: shoppingList[1] ="chocolate"
        
        function: del shoppingList[0]
        
        add: array1 =[1,2,3]; array2 = [4,5] array3=array1+array2 = [1,2,3,4,5]
        
        len(shoppingList) = 2
        
        max & min: max(array1) —— ASCII
        
        append: shoppingList.append("brocolli")
        
        count
        
        create an empty list: x= []
      - dictionary: unordered
        
        new(key:value): student={"Eric":14, "Bob":12}
        
        delete: clear(): remove all items /del dicName: delete the dictionary del d[key]
        
        get: len()/ keys() - values()/ d[key]
        
        update: d[key]=value; / update([dic || other])
      - tuples: ordered/immutable(cannot append/delete)
        
        create: tup = ("Math", 23,"Cats")
        
        del tup.
        
        application: location refresh every second
        
        Indexing： tup[1]: get one item;
        Slicing： tup[start index: n] : get n items from start position
    - - Multiple declaration: a,b,c = "", "","" || a=b=c=""
      - age="aa" || age=11 || age="aa","bb"
      - Wrong: var 2 = "";
  - - - if 5>2: print("aa") else: print(" ")
      - Nested If/Else: notice indentation
      - if 5>2: print("aa")
        elif a>1: print("2") else: print(" ")
    - - ≥/≤： greater/less than equal to; </> less/greater than; == equal to equal to； != not equal to;
    - - and/or
        
        if 5>2 and 4>3 : print("aa")
    - - for
        
        for i in range(0,5):
        ------print(i)
        
        range(start,end): give i value from 0 to 5(not include)
        
        range(start,end,n): counting by n[can be negative: decrease]; start, start+n, start+2n,...
        
        iterate the list/dictionary/tuple
        
        for i in shoppingList: ...
      - while
        
        while counter<10: ...
        
        Example: prime number:素数
        
        i%j: i is divisible by j
    - - break
      - continue
      - pass: filler statement
        
        if ...:
        else:
        ------pass;[else not do anything]
  - - - variables are too long: if a \
        == \
        b:
      - a = '''print(''hi there')
        print("cc")'''
- - - - global
      - local
      - ex: total =2
        def multiply(a,b):
        total = a*b;
        return total
        
        multiply(2,3) > 6;
        >>> total > 2
  - - - ex: help(string.upper())
    - - ex: dir(["aaaa"])
    - - eval("print('hello')") or eval("3*4")
    - - a = '''print("aa")
        print("cc")'''
        exec(a)——execute a
    - - string > number
        
        int(string) / float(string)
      - number > string
        
        str(number)
    - - age = input("enter your age:")
- - - - customer defined
        
        def displayStudent(self):
        return "student" + self.name
      - attributes related:
        not used frequently
        
        hasattr(instance, "attributeName")
        
        ex: hasattr(student, "age")
        
        setattr(instance,"newAttribute", "attrValue"):
        for special instance
        
        ex: setattr(instance,"grade","4th")
        
        getattr(instance,"attributeName")
        same result as instance.attributeName
        
        getattr(instance,"age")
        
        delattr(instance,"attributeName")
  - - - child class can invoke functions and variables in parent class.
        it can also change the value of parent variables.
    - - the same function as the parent class, the instance of child will invoke the overrided function in child class
- - - - file1 = open("fileName[c:\zz\zz.txt]", "r/w/x/a/b/t...")
        Notice:
        1.create/read/write/append mode must have only
        one.
        
        if the doc is not exist, python will create the doc automatically
        
        'a' open for writing, appending to the end of the file if it exists
        
        'b' binary mode
        
        't' text mode (default)
        
        'r' open for reading (default)
        
        in r mode, python will not create a new file if the file name is not exist.
        
        'w' open for writing, truncating the file first
        
        '+' open a disk file for updating (reading and writing)
        
        ex: open("","a+")
        can read and append simultaneously
        
        'U' universal newline mode (deprecated)
        
        'x' create a new file and open it for writing
    - - file1.read([number]): read all or specified number
        
        Notice: the first read() will move the cursor to the end of the doc, so the second read() cannot read anything.
      - file1.tell(): the current position[字符数] of the cursor
      - file1.seek(offset[,whence]):
        move the cursor to the specified position;
        
        offset: 偏移量；
        
        positive value: to the end of the file;
        negative value: to the start of the file.
        
        whence(optional): the start position of the offset,
        
        0-from the beginning(default),
        1-from current position,
        2-from the end of the file
    - - file1.write("contents")
        Notice:rewrite the whole file
      - appending to a file
        
        a mode
        file1.write("append contents")