File writing does not happen when it is supposed to happen in the program flow

This is not a new problem for me. From C to PERL to Python on Windows Mobile, Windows XP and other Windows versions this problem persists and f**ks my nerves.

Now in my lates开发者_StackOverflow中文版t script it again happens. To be more concrete: I have coded in Python a trivial script. Now the script writes correctly to the file when run from the debugger, but out of the debugger it does not work correctly. It does not write to file when it should. I am using python 2.6 with eclipse and pydev.

This is the code

import httplib2
import thread

ht = httplib2.Http();
list = []
k = 0

def check(proxy, port):
    global list
    global k
    try:
        head = ht.request(proxy, 'HEAD')
    except:
        return
    k = k + 1
    list.append(proxy)
    list.append(port)


def OnListCaller(ProxyList, OutFile, NListLen):
    global list
    global k
    filei = open(ProxyList, 'r')
    fileo = open(OutFile, 'a')

    while 1:
        proxy = filei.readline()
        if not proxy: continue
        port = filei.readline()

        proxy = proxy.rstrip()
        port = port.rstrip()

        thread.start_new(check, (proxy, port,))

        if k >= NListLen:
            for t in list:
                fileo.write(t + "\n")
            list = []
            fileo.close()
            fileo = open(OutFile, 'a')
            k = 0


OnListCaller('C:\proxy\input.txt', 'C:\proxy\checked.txt', 1)   

The problem is in the OnListCaller at the if k>=NListLen. The file should be updated when k is >= then a given value. Thanks to everyone.

Remember what your mom taught you:

always flush()

(in python, file_object.flush() followed by os.fsync(file_object.fileno()))

Regarding the code: It looks like the actual problem is one relating to threads, not to files:

Whilst you are executing this code:

    for t in list:
        fileo.write(t + "\n")
    list = []
    fileo.close()
    fileo = open(OutFile, 'a')
    k = 0

list is being modified by the threads you have spawned. I don't know the details of how 'for x in y' works with threads, but I imagine it is missing out the elements added to the list after the body for loop has been executed the first time.

To solve this you need a mutex for list which you lock for the entirety of this for loop (until you have cleared the list), and which you lock whenever you are adding an item to the list.

import httplib2
import thread
import os
import sys


ht = httplib2.Http();

def check(proxy, port, OutFile):
    global list
    try:
        head = ht.request(proxy, 'HEAD')
    except:
        return
    fileo = open(OutFile, 'a')
    fileo.write(proxy+"\n")
    fileo.write(port+"\n")
    sys.stdout.flush()
    os.fsync(fileo.fileno())
    fileo.close()

def OnListCaller(ProxyList, OutFile, NListLen):
    global list
    filei = open(ProxyList, 'r') 
    while 1:
        proxy = filei.readline()
        if not proxy: continue
        port = filei.readline()

    proxy = proxy.rstrip()
    port = port.rstrip()

    #TODO: regleaza pentru unix cand o sa fie nevoie

    thread.start_new(check, (proxy, port, OutFile,))


OnListCaller('C:\proxy\input.txt', 'C:\proxy\checked.txt', 1)

This is the fixed code.

If you have opened the file with a file handle in Python, remember to close it when finished. eg

f=open("file")
....
f.write(....)
f.close()

Obviously, you forget to close the file upon finishing working with it. If you want to check the contents of the file before closing it, call the flush() method. Example:

file = open("hello.txt", "a")
file.write(...)
file.flush()      // force write on the disk
file.close()      // finished using the file, close it

Check you code, not all open files are closed in it.