135

As far as I know, Variable is the default operation for making a variable, and get_variable is mainly used for weight sharing.

On the one hand, there are some people suggesting using get_variable instead of the primitive Variable operation whenever you need a variable. On the other hand, I merely see any use of get_variable in TensorFlow's official documents and demos.

Thus I want to know some rules of thumb on how to correctly use these two mechanisms. Are there any "standard" principles?

Improve this question
1
  • 6
    get_variable is new way, Variable is old way (which might be supported forever) as Lukasz says (PS: he wrote much of the variable name scoping in TF) Commented May 8, 2016 at 19:56

4 Answers 4

Reset to default
92

I'd recommend to always use tf.get_variable(...) -- it will make it way easier to refactor your code if you need to share variables at any time, e.g. in a multi-gpu setting (see the multi-gpu CIFAR example). There is no downside to it.

Pure tf.Variable is lower-level; at some point tf.get_variable() did not exist so some code still uses the low-level way.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Thank you so much for your answer. But I still have one question about how to replace tf.Variable with tf.get_variable everywhere. That is when I want to initialize a variable with a numpy array, I cannot find a clean and efficient way of doing it as I do with tf.Variable. How do you solve it? Thanks.
71

tf.Variable is a class, and there are several ways to create tf.Variable including tf.Variable.__init__ and tf.get_variable.

tf.Variable.__init__: Creates a new variable with initial_value.

W = tf.Variable(<initial-value>, name=<optional-name>)

tf.get_variable: Gets an existing variable with these parameters or creates a new one. You can also use initializer.

W = tf.get_variable(name, shape=None, dtype=tf.float32, initializer=None,
       regularizer=None, trainable=True, collections=None)

It's very useful to use initializers such as xavier_initializer:

W = tf.get_variable("W", shape=[784, 256],
       initializer=tf.contrib.layers.xavier_initializer())

More information here.

Improve this answer

4 Comments

Yes, by Variable actually I mean using its __init__. Since get_variable is so convenient, I wonder why most TensorFlow code I saw use Variable instead of get_variable. Are there any conventions or factors to consider when choosing between them. Thank you!
If you want to have a certain value, using Variable is simple: x = tf.Variable(3).
@SungKim normally when we use tf.Variable() we can initialize it as a random value from a truncated normal distribution. Here is my example w1 = tf.Variable(tf.truncated_normal([5, 50], stddev = 0.01), name = 'w1'). What would the equivalent of this be? how do I tell it I want a truncated normal? Should I just do w1 = tf.get_variable(name = 'w1', shape = [5,50], initializer = tf.truncated_normal, regularizer = tf.nn.l2_loss) ?
@Euler_Salter: You can use tf.truncated_normal_initializer() to get the desired result.
48

I can find two main differences between one and the other:

  1. First is that tf.Variable will always create a new variable, whereas tf.get_variable gets an existing variable with specified parameters from the graph, and if it doesn't exist, creates a new one.

  2. tf.Variable requires that an initial value be specified.

It is important to clarify that the function tf.get_variable prefixes the name with the current variable scope to perform reuse checks. For example:

with tf.variable_scope("one"):
    a = tf.get_variable("v", [1]) #a.name == "one/v:0"
with tf.variable_scope("one"):
    b = tf.get_variable("v", [1]) #ValueError: Variable one/v already exists
with tf.variable_scope("one", reuse = True):
    c = tf.get_variable("v", [1]) #c.name == "one/v:0"

with tf.variable_scope("two"):
    d = tf.get_variable("v", [1]) #d.name == "two/v:0"
    e = tf.Variable(1, name = "v", expected_shape = [1]) #e.name == "two/v_1:0"

assert(a is c)  #Assertion is true, they refer to the same object.
assert(a is d)  #AssertionError: they are different objects
assert(d is e)  #AssertionError: they are different objects

The last assertion error is interesting: Two variables with the same name under the same scope are supposed to be the same variable. But if you test the names of variables d and e you will realize that Tensorflow changed the name of variable e:

d.name   #d.name == "two/v:0"
e.name   #e.name == "two/v_1:0"
Improve this answer

1 Comment

Great example! Regarding d.name and e.name, I've just come across a this TensorFlow doc on tensor graph naming operation that explains it: If the default graph already contained an operation named "answer", the TensorFlow would append "_1", "_2", and so on to the name, in order to make it unique.
2

Another difference lies in that one is in ('variable_store',) collection but the other is not.

Please see the source code: 

def _get_default_variable_store():
  store = ops.get_collection(_VARSTORE_KEY)
  if store:
    return store[0]
  store = _VariableStore()
  ops.add_to_collection(_VARSTORE_KEY, store)
  return store

Let me illustrate that: 

import tensorflow as tf
from tensorflow.python.framework import ops

embedding_1 = tf.Variable(tf.constant(1.0, shape=[30522, 1024]), name="word_embeddings_1", dtype=tf.float32) 
embedding_2 = tf.get_variable("word_embeddings_2", shape=[30522, 1024])

graph = tf.get_default_graph()
collections = graph.collections

for c in collections:
    stores = ops.get_collection(c)
    print('collection %s: ' % str(c))
    for k, store in enumerate(stores):
        try:
            print('\t%d: %s' % (k, str(store._vars)))
        except:
            print('\t%d: %s' % (k, str(store)))
    print('')

The output:

collection ('__variable_store',): 0: {'word_embeddings_2': <tf.Variable 'word_embeddings_2:0' shape=(30522, 1024) dtype=float32_ref>}

Improve this answer

1 Comment

And why is this relevant?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.