HomeImplementationNeural NetworksLow-Rank Factorization
Neural Networks

Low-Rank Factorization

CEO 72 views

The key idea behind low-rank factorization is to replace high-dimensional tensors with lower-dimensional tensors. One type of low-rank factorization is compact convolutional filters, where the over-parameterized (having too many parameters) convolution filters are replaced with compact
blocks to both reduce the number of parameters and increase speed.

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, SeparableConv2D

# Define input shape
input_shape = (64, 64, 3)  # For a 64x64 image with 3 color channels (RGB)

# Model with standard 3x3 convolutional filter
model_standard = Sequential([
    Conv2D(32, (3, 3), input_shape=input_shape, padding='same', activation='relu')
])

# Model with compact convolution using separable filters
model_compact = Sequential([
    SeparableConv2D(32, (3, 3), input_shape=input_shape, padding='same', activation='relu')
])

# Summary of both models
print("Standard Convolution Model Summary:")
model_standard.summary()

print("\nCompact Convolution (Separable) Model Summary:")
model_compact.summary()